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ABSTRACT 


The quality of software management has an affect on the degree of success or 
failure of a software development program, this statement has been argued successfully 
by Martin J. Machniak in his thesis Development of a Quality Management Metric 
(QMM) Measuring Software Program Management Quality. The QMM metrics can be 
used both to characterize the quality of software management and provide a template for 
improving software management performance. 

Technical Performance Measurement (TPM) in the most basic form is a plan of 
expected technical achievement in which the actual progress is compared with periodic 
measurements. However, the difference between the plan and the actual measures is a 
technical variance which can be considered good or bad, depending on the level of 
tolerance given in the requirements. TPM is breaking new ground in the development of 
various techniques for TPM where planning is integrated with cost, schedule, and 
program impact assessment. 

The author administered the QMM questionnaire to measure the perceptions of 
program managers that have the responsibility for software development programs within 
the U.S. Army. The author then gathered TPM data using an informal verification and 
validation of the same programs used for the QMM questionnaire, and compared the 
results and found an inconclusive correlation between them. 
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I. INTRODUCTION AND BACKGROUND 


A. PROBLEM 

The U.S. Army is faced with the challenge of what are the best possible 
management tools to use for developing a more responsive, and a more dominate combat 
force to meet today’s needs and all future threats. 

The U.S. Army is presently developing an advanced family of networked air- 
based and ground-based vehicles that are used in maneuver; maneuver support; and 
sustain program systems including manned and unmanned platforms. 

These systems are networked by a Command, Control, Communications, 
Computers, Intelligence, Surveillance, and Reconnaissance (C4ISR) architecture which 
includes network communications, network operations, sensors, battle command systems, 
manned/unmanned reconnaissance and surveillance capabilities to enable levels of 
situational understanding and synchronized operations. The vehicle platforms are a 
fraction of the weight of the current weapon systems, and are just as lethal and 
survivable. 

The lightweight and smaller sizes are critical to meeting the Army’s future force 
deploy-ability goal, of transporting vehicles using C-130 aircraft. The technical 
challenges are unprecedented plus the time constraints are formidable for this program. 

One of the major technical challenges is the development of a first-of-a-kind 
communication network. This endeavor includes developing data for 18 advanced 
systems, with 53 critical technologies, employing 157 complementary systems, and 34 
mi llion estimated lines of software code. 

Traditionally the U.S. Army usually allows only 5.5 years for development of a 
single major system (between program starts and the production decision). The programs 
are tasked to compress development time even though this U.S. Army system of systems 
is comprised of several systems including: the network; an Abrams Tank replacement; 
Bradley fighting vehicles replacements, and a Crusader replacement. 
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The U.S. Army has been given the challenge to proceed with the strategy of using 
a timetable where over 75 percent of the critical technologies are immature. If the U.S. 
Army assumes everything goes as planned, the program will begin production most likely 
before all of its systems have been demonstrated. This is the kind of strategy the Army 
plans on going forward with for production and fielding of its systems. 

The U.S. Army is now in the System Development and Demonstration (SDD) 
phase. The U.S. Army’s acquisition program was approved by the Defense Acquisition 
Board in May 2003. Also there has been designated a joint Services program with the 
Army and Marine Joint Program Office. On July 22, 2004, Army officials announced 
plans to accelerate the delivery of selected vehicles to the current force. The plan expands 
the scope of the program’s SDD phase by adding four discrete “spin outs” of capabilities 
at two year increments for the current forces. 

Spin out one will begin fielding in 2008 and consist of prototypes fielded to the 
Evaluation Brigade Combat Team (EBCT). Following successful evaluation, production 
and fielding of spin out one will commence in 2010. This process will be repeated for 
each successive spin out. By 2014, the EBCT will be equipped with all new core systems. 
Other Brigade Combat Teams will have selected embedded new capability. 

This is the Army’s strategy for the main modernization program in the 21st 
century. It will ensure that the Army retains the combat advantage in critical capabilities 
plus having net-centricity, mobility, and a more efficient use of material and personnel. 
When fielded to the force, the U.S. Army will have replaced 40 year old equipment 
designed to win against Cold War enemies. This effort will benefit the Army, Marine 
Corps, and Special Operations Forces, and the Nation as a whole. 

Since software development is a major part of this new U.S. Army system of 
systems, it is imperative that the software development be managed effectively in order to 
assure that the Army’s strategy is successfully implemented. Effective management of 
the software development, in turn, requires that the requirements for effective 
management be understood, measured and monitored. 
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B. SOLUTION 

The author has in the following sections, of this thesis examined U.S. Army 
software programs to determine how well Quality Management Metrics (QMM) correlate 
to Technical Performance Measurement (TPM). The author administered QMM 
questionnaire surveys to software Program Managers (PM) in software acquisition, and 
compared data from TPMs within the same programs. The thinking behind this research 
was to explore the data of TPM and QMM to see if there is a good working relationship 
between the metrics of each and how a software program might benefit the U.S. Army in 
managing these enormous developmental projects that can have tremendous political 
ramification and unwanted consequences. However, what the author did not do and has 
left for future research work was to address the relationship between Earned Valued (EV) 
and QMM. 

C. RESEARCH QUESTION 

This thesis focuses on answering the following question: 

1. How well does the quality management metrics (QMM) correlate to the 
technical performance measurement (TPM)? 

D. SCOPE, LIMITATION, AND ASSUMPTIONS 

This thesis describes how quality management metrics (QMM) correlates to the 
technical performance measurement (TPM). It has been argued successfully that the 
quality of software management can have an affect on the degree of success or the 
possible failure of a software development program. This argument was presented by 
Martin J. Machniak, his thesis developed metrics for measuring the quality of software 
management along four dimensions: requirements management, estimation/planning 
management, people management, and risk management. This QMM used in software 
development for program managers consists of a composite score obtained from a 
questionnaire administered to the program manager and their peers. The QMM reflects 
the success in the quality of software management, plus, it can be used as a template for 
possible improvement in software management performance. The author administered the 
same questionnaire survey to measure the conceptual performance of the individuals 
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responsible for Army software development programs on the government side of the 
house. The author also identifies, how the Technical Performance Measures (TPMs) are 
applied, and how (TPMs) are reported. The author will provide data how this process 
utilizes TPMs: (a) as key measures for indicators of whether or not a program is a 
success technically; and (b) in evaluating a program’s ability to meet requirements. TPM 
metrics are used to track and compare performance estimation, predictions, and actual 
measurements against specified and allocated goals over time. The author feels that the 
correlation between QMM and TPM can provide Program Managers (PM), Integrated 
Product Team (IPT) leaders, and customers, with good objective evidence in achieving 
design quality with approved requirements, and quality program management using 
QMM and TPM as tools for program success. 

E. METHODOLOGY 

The author is employed at the U.S. Army Tank Automotive, Research and 
Development Engineering Command (TARDEC). The author was placed on a 
developmental assignment to provide software quality engineering support to the 
developing combat systems in an Integrated Product Teams (IPT) in areas of Integration 
and Simulation Testing, Modeling and Simulation, and Training. 

The author conducted research in developing this thesis from various Army 
programs by a study of strategy used in the areas of Technical Performance Measurement 
(TPM), and administration of the Quality Management Metrics (QMM) questionnaire 
surveys. The surveys were given to the software development Program Managers (PM) in 
software acquisition, to determine if a correlation could be drawn between the two 
metrics. 

The major challenge encountered and overcome during the completion of the 
thesis was the consolidation of all information through the study of the various programs, 
plus research, and arranging interviews with very busy Program Managers working under 
tremendous pressure to do it right the first time. The internet provided the author with 
good reading material on the methodology in software project management and in 
software project management strategy in general for industry as a whole. The author 
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found that interviews provided insight to what worked, what didn’t work, and what was 
too costly to include in some projects. 

F. ORGANIZATION 

The chapters that follow describe what the author found during his study of the 
various programs, which included administration of the Quality Management Metrics 
(QMM) and Technical Performance Measurement (TPM). 

CHAPTER I: Introduction: problem, solution, research questions, scope, 
limitation, and assumptions, methodology and organization for the author’s thesis. 

CHAPTER II: The components of QMM, TPM and data from both metrics. 

CHAPTER III: Informal Verification and Validation. 

CHAPTER IV: Conclusions, Recommendations and Future Work. 
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II. METRIC METHODOLGY OF QMM AND TPM 


A. MOTIVATION 

The author felt that there was something missing in the software program 
management equation that might have been over looked in the current quest for cost- 
effective, high-quality software. The possible missing part may be the correlation 
between QMM and TPM. QMM has been proven by Machniak that the quality of 
software program management can be and is measurable, and available for input in 
costing and scheduling tools. The results can be provided to program managers so that 
they may pinpoint such areas in software program management where improvements are 
needed, and can be made. The capability to measure the quality of management of 
software projects objectively allows for accurate cost models where impact in 
management quality, including cost factors, will provide a means for software project 
management improvement using assessment by feedback and correction. 

Technical Performance Measures can be used to develop a plan of expected 
technical achievement to which the actual progress is compared using periodic 
measurements or tests. The TPMs are indication of compliance in design to requirements 
captured in specifications and to present management with quantitative data to determine 
whether action is required. The TPM approach, using various techniques of risk analysis 
and probability, offers a promising method that incorporates technical assessments, 
resulting systematically from technical parameter measurements to derive more discrete 
management data sufficiently early to allow for cost avoidance. Therefore, providing 
needed information that allows the managers enough time for making informed decisions 
on schedule, cost, and a review of technical requirements early in the program. 

In this thesis, the author examines the possibility of a correlation between TPM 
and QMM in each of the four areas covered in the QMM questionnaire survey. The 
author’s intent is to discover if any of the four areas of the QMM survey have a stronger 
correlation to TPM than others, in identifying contribution to management quality and 
possible project success. 
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B. STRATEGY 

The method developed in approaching the correlation between QMM and TPM 
included but was not limited too reviewing recommended practices, textbooks, on-line 
publications, and having interviews with various personnel from senior program 
managers to system developers. The QMM and TPM metrics measured the quality of 
management plus the technical performance on three specific software programs. 

The author’s goal is to draw an objective correlation between QMM and TPM to 
which program management can be compared and ranked thus giving a baseline for 
quantifying improvement. In the next few paragraphs an explanation will be given for 
QMM and TPM metrics. 

C. QUALITY MANAGEMENT METRICS (QMM) 

The QMM developed by Machniak in response to these concerns consists of 
various survey questionnaires covering these four areas: Requirements management; 
estimation and planning management; people management; and risk management, see 
Machniak thesis on QMM [REF 25]. 

The QMM survey is a questionnaire designed to be given to software project 
managers, and software developers who have global impact on software projects. 
Mackniak applied the survey on three software programs at the Space and Naval Warfare 
Systems Command initially, and then Grossman validated the QMM on ten software 
programs at Edwards Air Force Base, proving furthermore that there is a correlation of 
good quality management and the success of a software project. 

1. Requirements Management 

Software requirements management focuses on managing the process of 
extracting, developing, defining, and refining the requirements of a software program 
[REF 25]. It is the current belief that quality management of a program’s requirements 
must have established procedures and structure to ensure that requirements specifications 
are complete; consistent; understandable to the reader; lacking ambiguity; having a 
known origin; and not having vague design stipulations [REF 25]. Also, requirements 
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need to present one idea per requirement, and address the requirement attributes. Good 
requirement management provides current status by tracking the dates, versions, 
relationships to other requirements, and the priority rationale behind such decisiveness. 

2. Estimation/Planning Management 

The use of software estimations are basically one of the main methods in which 
planning is performed in software programs. The QMM estimation/planning management 
section will not give a specific estimation technique as being the right one over others 
used. However, the QMM estimation/planning management section will seek to quantify 
management’s efforts in the estimation/planning process. In other words the questions in 
the survey are used to determine if the choice of a technique is appropriate and how well 
that technique is implemented in the program. 

3. People Management 

In QMM quality people management covers the need for organizational 
management providing a good atmosphere with proper working conditions with all 
environmental efficiencies maximized. 

In QMM, quality people management has the work flow aided by delegation and 
task ownership with management monitoring those activities and processes. Questions 
QMM ask: Are the roles well defined for all team members? Do the team members’ have 
a part in the project planning and decision making process? Is there effective 
communication being given from top down, and bottom-up with good customer or team 
communication occurring? Also, it would be best that the program managers have a good 
working knowledge in the technology being managed. 

4. Risk Management 

The QMM references a proactive approach on quality risk management. The use 
of a formal risk management plan is developed usually before the program begins with a 
list of risks identified by the team members, and customers through assessments and the 
use of checklists. Throughout the life of the development risks are assessed and tracked 
by management. The prioritization of risks is based on the probability of occurrence and 
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negative consequences. A risk strategy is formulated to mitigate risks with a plan 
developed to allocate needed resources in reflection of risk priorities. Risk data is to be 
shared [REF 25]. 

The author notes that the QMM was developed to reflect the management needs 
of large projects and tests through the questionnaire survey, and formal methods are used 
for the management of requirements, performing estimations and planning, managing 
people, and managing risk. 

D. QMM SURVEYS 

Software program managers on software development programs at U.S. Army 
TACOM were asked to complete the QMM survey. These individuals were selected 
because of their complete understanding of the program and the fact that they had a good 
understanding of the management practices which were implemented throughout the 
software program. The software program managers used a specific point in time in the 
program for the evaluation of the program management, so that the individual team 
members were able to identify the selected point in time and evaluate the program. Also, 
the TPM evaluation was selected during the same point in time. 

In the best interest of the program and to maintain confidentiality of the survey 
the programs are identified as programs A, B, and C. 

The interviewees, after completing the survey, were asked to rate the success of 
that period or selected point in time using an evaluation scale of zero to ten. The score of 
zero corresponded to program failure and ten corresponded to a completely successful 
program. A score of ten meant that the software program produced a product on time, 
and within the budget allocated as well as complete customer satisfaction with the quality 
product. 

Part I of the QMM questionnaire survey: 

This part of the survey questionnaire is the pair-choice questions. It consists of 
two questions placed side by side on a single line within a column next to each question. 
The interviewees were asked to check a box next to the question or statement which 
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closely reflected what was happening on the evaluation program at the specific point in 
time. The interviewees made choices of the two for each line of the survey questionnaire. 
The question or answer that most likely reflected the evaluation program need not be an 
exact match. There were two different ideas for each pair-choice question. This was 
done in an effort to find a tendency of the interviewee in the area of interest by way of 
formal requirements documentation versus informal requirements or documentation. 
Most often the pair-choice questions were repeated with different wording to confirm the 
earlier choices and measure the strongest tendency. The format of the questionnaire 
survey using the proper mix of questions, plus a variation with repetitions, was designed 
to reach a consensus on issues, measure tendencies, and show strengths [REF 25]. 

Part II of QMM Questionnaire Survey: 

This part of the survey questionnaire is basically yes or no questions that consist 
of one question per line with three columns next to it giving the person a possible “yes,” 
“no,” and “N/A” answer [REF 25]. 

The interviewee answering the questions, would answer as it pertains to a 
program manager and the program during a specific point in time on the program by a 
“yes,” “no,” and “N/A” in the box next to each question, with the use of the “N/A” box 
discouraged unless the program manager has no say in such issues. 

In the requirements management pair-choice section, a score of zero to two is 
possible having different upper bounds on the score of each question. This is based upon 
the relative weight and importance of each question in the section. However, in the 
estimation/planning management, people management and risk management sections the 
possible scores were zero to one. 

The questions that answer either yes or no have a score that ranges from minus 
four to plus four. This score is based upon its relative weight and importance in the upper 
and lower bounds of the survey questionnaire. 

This was determined by Machniak [REF 25] and stated as such [Q,M,M&G]. The 
QMM equation is given by: 
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QMM=0.92RQM+0.67EPM+0.55RKM+1.86PM, where: 


RGM is the requirements management metric, 

EPM is the estimation/planning metric 
RKM is the risk management metric; 

PM is the people management metric 

Having coefficients ranging from 0.92, 0.67, 0.55 and 1.86 as the importance 
coefficients of the requirements, estimation/planning, risk and people management 
metrics respectively. As the importance coefficients have been determined through focus 
groups, interviews with one-on-one experienced software professionals [REF 25]. 


Data Analysis 


Program 

Program A 

Program B 

Program C 

Participant 

ApM 

A, 

B pm 

Bi 

CpM 

c, 

QMM score 

509.65 

522.63 

569.03 

559.44 

314.83 

229.21 

QMM percent 

77.35 

79.32 

86.36 

84.91 

47.78 

45.36 

Success score 

8 

8 

9 

8 

6 

6 

Mean success score 

(0-10) 

8 

8.5 

6 


Table 1. Results of Informal QMM Validation 


Table 1 is the summary of the three programs included in this analysis using data 
from the program manager and independent development team members. In all of the 
following tables QMM percentage score, requirements management, estimation/planning 
management, people management, and risk management scores have all been adjusted to 
a scale of zero to ten. The zero score in Table 1 corresponds to zero percent of the points 
found possible in the section where as a score of ten corresponds to a possible 100 
percent or 100 points in the section. 


Program 

PM 

Program 

Score 

PM 

QMM 

Score 

PM 

Requirements 

Management 

PM 

Estimation 

Planning 

PM 

People 

Management 

PM 

Risk 

Management 

A 

8 

7.7 

88 

66 

169 

56 

B 

9 

8.6 

101 

80 

193 

64 

C 

6 

4.8 

49 

70 

6 

60 


Table 2. Program Manager Summary Data 


12 





Table 2 is a summary of the Program Manager Data. The first column from left 
to right identifies the program, the second column provides the program managers 
subjective program score, the third column provides the QMM score based on the 
program managers questionnaire survey, while columns four through seven provides the 
program managers QMM requirements management, estimation and planning 
management, people management and risk management scores reflecting the 
questionnaire survey. 

E. TECHNICAL PERFORMANCE MEASUREMENT (TPM) 

Technical Performance Measurement (TPM) is, in its most basic form a plan of 
expected technical achievement to which the actual progress is compared using periodic 
measurements or tests, see [REF 20]. 

Technical performance measures are engineering and physical measures, such as 
computer throughput, radar detection range, number of possible users and programmatic 
metrics used by a program in gauging effectiveness in developing designs to ensure that a 
design meets the performance specified by the customer. The TPMs are indicative of 
compliance in design to requirements captured in specifications and presents 
management with quantitative data to determine whether action is required. The TPM has 
been integrated with requirements management issue and action management, baseline 
management, and risk management. TPMs evaluate the adequacy in evolving solution 
through engineering changes and trade studies to identify deficiencies that impact the 
systems ability to meet the performance requirement. Technical characteristics are 
evaluated to identify problems through engineering analyses and should indicate if 
performance is being met as specified in contracts or other requirements. As the system 
concept is being developed the TPMs are initially defined, and are formalized during 
contract start through requirements definition. Existing TPMs can be modified per 
program needs, and new TPMs can be added to the system at any time during the 
program. 

Technical Performance Measurement (TPM) supports the Army’s objective and 
strategy by establishing adaptive and affordable processes. It also supports the monitor 
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and control process area in the system engineering process. In the government systems 
program managers and their teams, find themselves in an environment that creates 
pressures that can be translated into products being delivered using best value analysis 
with cost as the overriding determinant. (The TMP approach, using various techniques of 
risk analysis and probability, offers a promising method that incorporates technical 
assessments, resulting systematically from technical parameter measurements to derive 
more discrete management data sufficiently early to allow for cost avoidance.) Therefore, 
this, in essence, provides needed information that allows the managers more time for 
making informed trade-off decisions early in the program. 

A few recent initiatives are breaking new ground in the development of 
sophisticated techniques for TPM planning, integration with cost and schedule, in such 
manner to be reflected in Earned Value Management (EVM) data. 


Earned Value Management: 

a. Earned Value Management (EVM) is an integrated system of project 
management and control which has enabled the contractor and their customer 
to monitor the progress of a project in terms of integrated cost, schedule and 
technical performance measures, see Appendix B. 


Integrated Product Team (IPT) Ownership: 

b. EVM system is created, owned and managed by the Prime Contractor, but the 
customer has full and timely visibility of the information at any time. From 
this perspective this means that there is greater equality of information 
between the contractor and the customer which is fundamental to true 
partnering. 

EVM function: 

c. EVM provides a reference point which is an objective view of the status of the 

contract such that the value to the end or goal reflects work completed to date. 

This needs to be compared with both the planned expenditure and the actual 

14 



costs to determine the performance to date and to give early indications of 
problems. Now EVM may also be used to enhance cost forecasting, risk 
management and as the basis for payment against the contract. 


EVM data requirement: 

d. The way in which EVM is implemented, the contractor must have a validated 
system that can accurately measure the following three fundamental factors: 

1. The Budgeted Cost of Work (BCWS) or what is known as planned costs. 

2. The Actual Cost of Work Performed (ACWP) or what is known as the 
actual cost of progress made. 

3. The Budgeted Cost of Work Performed (BCWP) or earned value. 


Earned Value Management system: 

e. The heart of EVM is the Work Breakdown Structure (WBS). The WBS is a 
product oriented family tree structure of all of the goods and services to be 
built or supplied. The WBS is a consistent and visible framework that 
displays and defines the products as elements that relate to the end product. 

f. The WBS needs to be defined down to at least the level at which EVM 
reporting will be applied, and within a WBS that adequately meets their data 
requirements. 

g. The schedules that are produced for the lower elements of the WBS should be 
planned to the greatest possible detail in that the resulting activities are of 
manageable duration and can be assigned to a single part of the organization. 

h. Earned value is based on assigning a value at the activity level to the 
achievement of project work. Ideally, to determine non-subjective 
achievement such as milestones and deliverables, one would need to have 
them based on the planned cost (in money or hours) for achieving the goal. 

i. The control sometimes called Cost Account (CA) coincides with the level at 
which EVM reporting will be applied. The CA has a dedicated manager 
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appointed. This individual is empowered to plan and deliver within time and 
cost, those constraints set forth within the CA. 

EVM is generated: 

j. This begins once the project is underway -the contractor will start to earn 
value by the commencement and completion of individual activities. The 
summation of the values earned in a particular control account gives the 
earned value of the CA to date. 


EVM data is presented: 

k. The earned value is plotted against the planned and actual costs over time. 

This is a very clear way to show the status of the project. The progress report 
has a basic tabulation of the three basic data elements, the estimate at 
completion and the budget, and their derived data elements, or variances, 
which are measured in terms of resources such as man-hours or cost. The 
derived data elements are: 

1. Cost Variance (CV) - The difference between the planned and actual 
resource usage for an element of work. A negative variance means that 
more money was spent for the work accomplished than was planned. Cost 
Variance is obtained by comparing actual cost with earned value: 

2. Cost Variance = Earned Value - Actual Cost 

3. CV = BCWP - ACWP 

4. Schedule Variance (SV) - The difference between the budget and the 
earned value for an element of work is called the schedule variance. 

5. Schedule Variance = Earned Value - Budget 

6. SV = BCWP - BCWS 

7. Variance at Completion (VAC) - The difference between the total budget 
allocated for a piece of work and the project manager’s estimate of the 
actual resource cost at completion. 
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An example of the way EVM data may be presented in the form of a graph is 
shown in Figure 1. This can be available at the total contract level and at all WBS levels 
down to the lowest level set within the contract. 


Time Planned 

‘now' completion 



KEY 

EAC Estimate at Completion 
BAC Budget at Completion (current) 

BCWS Budgeted Cost of Work Scheduled (current) 

BCWP Budgeted Cost of Work Performed (earned value) 

ACWP Actual Cost of Work Performed 

Figure 1. EVM Graphical Representation 
EVM data in graphical representation: 

l. These graphical representations are useful management information tools. For 
example, in Figure 1, the graph may represent a project or task that appears to 
be underachieving in terms of both cost and schedule. Now if corrective 
action is not taken, the project/task will be completed behind schedule and 
over budget. 

m. As well as the derived performance indicators mentioned above, there are two 
measures of efficiency which are also useful for determining the status of the 
project: (a ratio of less than one implies that work is underachieving against 
the plan, and above one implies better than the plan). 

1. Cost Performance Index (CPI) = How much it really costs to earn one 
pound of budget or the “value for Money” report. 

2. Cost Performance Index = Earned Value / Actual Cost 

3. CPI = BCWP / ACWP 
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4. Schedule Performance Index (SPI) - The Schedule Performance Index is 
the ratio of Earned Value and the Planned Achievement. 

5. Schedule Performance Index = Earned Value / Budget 

6. SPI = BCWP / BCWS 

Reporting Cycle: 

n. The reporting cycle should as a minimum tie in the contractors and customers 
internal accounting periods (usually monthly - although on the high risk 
projects the reporting cycle is weekly). 

Cost & Schedule Performance Index Chart: 

o. The CPI/SPI trend chart in (Table 3) provides a summary of the three 
programs A, B and C that are included in this analysis using data from TPM 
presented though EVMS: 

1. SPI = (BCWP/BCWS) and CPI = BCWP/ACWP. 

2. The (BCWS) Budgeted Cost for Work Scheduled, which is distributed 
cost for all the work that is realistically time-phased based on schedule, 
scope and resources. 

3. The (BCWP) Budgeted Cost for Work Performed; also referred to as 
Earned Value. The budgeted cost for all the work actually accomplished in 
a given period of time, as a measurement of work progress. 

4. The (ACWP) Actual Cost of Work Performed; also referred to as (ACI) 
Actual Cost Incurred and can mean actual cost as recorded in the 
accounting system for all the work associated with a given period of time. 
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Prog 

Mth. 

OCT 

NOV 

DEC 

JAN 

FEB 

MAR 

APR 

MAY 

JUN 

JUL 

AVG 

COE 

A 

SPI 

98% 

100 

% 

100 

% 

121 

% 

117.5 

% 

114.5 

% 

100% 

100% 

100% 

100% 

105 

1.05 

CPI 

108.5 

% 

107 

% 

101.5 

% 

123.5 

% 

113.5 

% 

114% 

113.5 

% 

113.5 

% 

113.5 

% 

113.5 

% 

112.5 

1.13 

B 

SPI 

100 

% 

100 

% 

100 

% 

100 

% 

100% 

100% 

100% 

100% 

100% 

100% 

100 

1.00 

CPI 

100 

% 

100 

% 

100 

% 

100 

% 

100% 

100% 

100% 

100% 

100% 

99.9 

% 

99.99 

1.00 

C 

SPI 

98.5 

% 

98.2 

% 

98.2 

% 

98.2 

% 

97.1 

% 

97.1 

% 

96% 

97.5 

% 

97.5 

% 

97.5 

% 

97.37 

.974 

CPI 

95% 

96.2 

% 

96.1 

% 

97.1 

% 

101% 

100.5 

% 

104% 

104.5 

% 

105% 

104.5 

% 

100.2 

1.00 


Table 3. SPI/CPI Trend Chart 


In Table 3, the author took the average of each program’s SPI and CPI over the 
ten month period, and then the coefficient. 

F. SUMMARY: 

In section II the author presented the metrics that are to be used to answer the 
question... “Is there a correlation between Quality Management Metrics (QMM) and 
Technical Performance Measurement (TPM).” 

1. The quality of software management has an effect on the degree of success or 
failure of a software development program, this statement has been argued 
successfully by Martin J. Machniak in his thesis Development of QMM 
Measuring Software Program Management Quality. The QMM metrics can 
be used both to characterize the quality of software management and provide a 
template for improving software management performance [REF 25]. 

2. TPM uses engineering data that physically measures: computer throughput, 
radar detection range, number of users and other programmatic metrics such 
as EVMS. This helps the program manager gauge the effectiveness of a 
developing design in meeting the performance specifications developed for 
the U.S. Army. The TPM is used as an indicator for compliance of a design to 
requirements or specifications and presents management with quantitative 
data that can be used to determine if corrective action is needed. TPM is 
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integrated with EVMS reflecting cost and scheduling, design requirements, 
issue and action management, baseline management, and risk management for 
impact assessment [REF 20]. 

The author administered the QMM questionnaire to measure the perceptions of 
program managers from programs A, B, and C that have the responsibility for the 
software development within each of the said programs for the U.S. Army. The author 
then gathered TPM data using a metric methodology from the same programs given the 
questionnaire, and developed the data tables for possible correlation between them if any. 

In Section III the author presents the informal verification and validation. 
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III. INFORMAL VERIFICATION AND VALIDATION 


A. MOTIVATION 

The methodology and structure for evaluating the possible correlations between 
Technical Performance Measurement (TPM) and Quality Management Metrics (QMM) 
have been laid out in the previous chapter, with the informal verification and validation 
presented in this section. 

Informal verification and validation being necessary ensuring that both metrics 
TPM/EVM and QMM reflect a positive correlation between each other having measured 
the cost and scheduling with TPM/EVM and the quality of software program 
management in a fashion as accurately as possible using QMM. 

B. STRATEGY 

The verification and validation approach is informal. The evaluation was of three 
software programs using the QMM survey score and the TPM/EVM metrics from the 
same three programs during the same time period. The program manager and one 
program developer from the same team evaluated program A, and such was the case for 
programs B and C, using the program manager and one program developer. 

In developing a frame of reference for which a correlation can be established from 
the QMM survey results, two measures were used. The two measurements are the 1) 
QMM percentage score, and 2) the overall program success score. 

1. The QMM percentage score is derived by first taking the surveys minimum 
QMM score and normalizing it to zero. This can be done by adding 130.86 to 
the minimum score of -130.86 in doing so makes it zero. The maximum 
QMM score possible from the survey is 528.00, adding 130.86 for 
normalization, the survey maximum possible score is now 658.86. 

2. The QMM percentage score is obtained by dividing the minimum normalized 
score by the maximum normalized score, and multiplying the results by a 
hundred. 

The equations taken from Martin J. Machniak thesis: 
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QMM (min) + 130.86 = 0.00 = QMM (min normalized) 

QMM (max) + 130.86 = 658.86 = QMM (max normalized) 

QMM (score) + 130.86 = QMM (score normalized) 

(QMM score normalized / QMM max normalized) X 100 = QMM % score. 

The participant taking the survey assigns an overall program success score based 
on how they feel the program is doing from their perspective and is totally subjective. 
However, for the most part the success of a program is measured by the final product 
performance and whether or not it meets the user’s satisfaction and the stakeholder’s 
expectation. 

A comparison is made between the QMM survey score and the individual overall 
success score, and to the mean overall success score of the program. 

The mean overall success score of the program is based on surveys from the 
project manager and other individuals capable of judging the overall success of the 
program. The scale used to measure the overall success of a program by the individual 
taking the survey is presented by a score from zero to ten. The best program would be 
given a score of ten, with a zero score being a program failure. However, the author 
would like to make it clear that an overall success score of ten is defined as having 
perfect software product and program execution, and that success or failure of a software 
program is not always due to the actions of program management. 

The comparison of the three, QMM percentage score, the individual score, and 
the mean overall success score of the program will establish any correlation between 
them for each program. The example, given by Martin J. Machniak in his thesis, dated 
December 1999, stated that the possible overall success score of seven corresponded to a 
QMM percentage score of 70 percent plus or minus 5 percent would indicate a strong 
correlation. An overall success of seven to QMM percentage greater than a plus or minus 
five percentage points of 70 percent, and less than plus or minus 15 percentage points of 
70 percent can be considered a fair correlation. However, in a program where 8 is the 


22 



overall success score, with relationship to a QMM percentage score of 40 
percent, the correlation is considered weak. 

The Technical Performance Measurement (TPM) metrics was evaluated based on 
the Earned Value Management (EVM) data which is an integrated system to monitor the 
progress of a project in terms of integrated cost, schedule and technical performance 
measures. The author would like to note that traditional project management practices 
tend to compare the actual costs with planned expenditures, and sometimes confuses 
actual known costs with actual known progress. In as much as actual costs are not 
necessarily in some cases good measures of progress, the EVM can provide a third 
reference point which is an objective view of the project status; an example would be the 
value to the end goal of the work completed to date. Using EVM, problems can be 
indicated early by comparing both the planned expenditure and the actual costs to 
determine the performance to date of the project. 

The project manger, in order to implement EVM, needs to have a validated 
system that accurately measures the: 1) planned cost of work, also known as the 
Budgeted Cost of Work Scheduled (BCWS); 2) the actual cost of the progress made, also 
known as the Actual Cost of Work Performed (ACWP); 3) The earned value, also known 
as the Budgeted Cost of Work Performed. 

The author states that the Work Breakdown Structure (WBS) provides a sort of 
family tree where all the goods and services are to be supplied. This family tree gives a 
visible framework to display, and define the products and elements that make up the end 
product. Ideally Earned Value is assigned value at an activity level to an achievement of 
project work; and is non-subjective; based on milestones; deliverables; and based on 
planned costs, such as money or hours of achieving that milestone or deliverable. The 
Earned Value techniques are numerous and can be applied to various activities with the 
guidance from a specialist in that particular activity. 

The author was given TPM/EVM data that was available from Control 
(sometimes called Cost) Account (CA) in programs A, B & C. The programs appointed 
managers from each program A, B and C provided two indicators: 
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1. Cost Performance Index (CPI) which is how much it really costs to earn one 
pound of budget or the “Value for Money” report: Cost Performance Index = 
Earned Value/Actual Cost, CPI = BCWP/ACWP. 

2. Schedule Performance Index (SPI) that shows the schedule Performance 
Index as the ratio of Earned value and the Planned Achievement: Schedule 
Performance Index = Earned Value/Budge, SPI = BCWP/BCWS. 

C. RESULTS 

In the following paragraphs are the results of the QMM surveys and the 
TPM/EVMS data. 

1) The scores form the QMM survey presented in Table 4 summary for the A, B, 
and C programs. The QMM was determined for each of the three programs A, B, and C 
using QMM score as a percentage of the QMM maximum possible score of each 
program. The percentages of each program were compared to the scores given by survey 
participants for a comparison. This provides a mean success score for each of the 
programs too include both the Project managers and other associates within each program 
that have the insight forjudging program success. 


Program 

Program A 

Program B 

Program C 

Participant 

ApM 

Ai 

B pm 

Bi 

CpM 

Ci 

QMM score 

509.65 

522.63 

569.03 

559.44 

314.83 

229.21 

QMM percent 

77.35 

79.32 

86.36 

84.91 

47.78 

45.36 

Success score 

8 

8 

9 

8 

6 

6 

Mean success 
score (0-10) 

8 

8.5 

6 


Table 4. 


QMM Results Summary Comparison. 


The author notes that the survey results for all programs reflect a correlation 
between the QMM percentage ranking, the overall success ranking of the program, 
individual success ranking scores, and the mean ranking scores. 

All QMM survey summary sheets for programs A, B, and C are enclosed and 
presented in Appendix C. 
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a. The examination of the survey summary sheets for program A, found that 
there was a slightly lower score in the areas of people and requirements 
management. But, the end product was good due mostly to experienced 
personnel with a history of working together both as stakeholders, users and 
technical staff. 

b. The examination of program B reflected very good scores in all four areas of 
the QMM survey, and provided an excellent product with a timely delivery. 
Once again the program had experienced personnel in all areas of 
requirements management, people management, risk management, 
estimation/planning management, and supported by an excellent technical 
staff. However, the author would like to point out that a program where 
people are this experienced may have the attitude that they have seen it all 
before and the Project Manager needs to have very strong leadership skills 
with a reputation of known success in order to guide them. 

c. Program C scored poorly in two areas of the QMM survey, and this was 
reflected in the QMM score, QMM percentage score, the success score given 
by the participants, and the mean success score. The first was requirements 
management, and second was people management. In requirements 
management the problem issues stem from not having very well defined 
technical goals and constant changing program requirements. The changes 
made in the technical goals without communicating with all the stakeholders 
and the users left the technical part of the program unable to establish TPM’s 
or EVM’s. The lack of well defined requirements and immature technology 
caused personnel to request a transfer from the program. The level of 
frustration in all areas of the program made the turnover of personnel very 
common and to the point where training was done by new hires for other new 
hires. 

The author found the self enhancement bias stated in Martin J. Machniak thesis to 
be true. In all interviews with program managers most felt that they could always solve 
the other program managers program problems. But, when asked about their own 
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program problems stated simply theirs were unique only to their program. The need for a 
QMM survey administrator to explain the intent is a must, and, interviewing the people 
before and after the QMM survey is necessary in order to have everyone become aware 
of the differences as perceived in what is thought to be happening in a program and what 
is actually taking place and what is required in the program. The after actions review with 
all participants of the survey discussing questions and answers, for the most part was the 
biggest benefit of the QMM process. All QMM summary sheets for programs A, B, and 
C for all survey participants are found in Appendix C. 

All copies of the completed survey from each of the three program managers and 
other program participants are included in Appendix A. Also, QMM survey questionnaire 
templates with points and ranking of each response can be found in Appendix A. 

2) The data from the TPM/EVM is presented in Table 5 summary for A, B, and C 
programs. The CPI and the SPI average plus, coefficient of each program over a ten 
month period included in this analysis using data from TPM presented though EVMS. All 
EVM summary sheets for programs A, B, and C are enclosed and presented in Appendix 
C. 

The EVM performance goal was determined for each of the three programs A, B, 
and C using a ratio of less than one implies that work is underachieving against the plan, 
and above one implies better than the plan. 


Prog 

Mth. 

OCT 

NOV 

DEC 

JAN 

FEB 

MAR 

APR 

MAY 

JUN 

JUL 

AVG 

COE 

A 

SPI 

98% 

100% 

100% 

121% 

117.5 

% 

114.5 

% 

100% 

100% 

100% 

100% 

105 

1.05 

CPI 

108.5 

% 

107% 

101.5 

% 

123.5 

% 

113.5 

% 

114% 

113.5 

% 

113.5 

% 

113.5 

% 

113.5 

% 

112.5 

1.13 

B 

SPI 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100 

1.00 

CPI 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

99.9 

% 

99.99 

1.00 

C 

SPI 

98.5 

% 

98.2 

% 

98.2 

% 

98.2 

% 

97.1 

% 

97.1 

% 

96% 

97.5 

% 

97.5 

% 

97.5 

% 

97.37 

.974 

CPI 

95% 

96.2 

% 

96.1 

% 

97.1 

% 

101% 

100.5 

% 

104% 

104.5 

% 

105% 

104.5 

% 

100.2 

1.00 


Table 5. 


CP 


& SPI Results Summary Comparison 
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The author reviewed the data from Table 2, which provided a summary of the CPI 
and the SPI average plus, coefficient of each program. Earned value is a means of placing 
a dollar on project status, in this way provides project manager’s a way to compare 
budget versus actual costs versus what the project status is in dollar amounts. In order to 
have a proper analysis of the project the following items will be needed: budget, earned 
value, actual costs, and forecasts. A reference to Figure 2 indicates it is without earned 
value, it shows actual costs as less than what has been budgeted, and it is impossible to 
tell if the actual costs are less or if work is progressing at a slower rate than planned or 
actual costs are less than what was budgeted. Earned value can be defined as the sum of 
the budgets for the work that is complete, and earned value for completed project 
activities is equal to the total budget. However, for activities not started, the earned value 
is equal to zero. Objective judgments or Performance Measurement Techniques (PMT) 
refers to multiplying the budget by the percentage complete to get the earned value. The 
author notes that work performed by a project manager and quality control inspector is 
referred to as “level of effort” and value is as budgeted. Plus, as long as the task is 
completed the value is earned. Figure 2 gives the Schedule Variance (SV) minus the 
difference between the earned value and the budget minus the Cost Variance (CV) minus 
the difference between the earned value and the actual costs. 

D. TPM/EVM DATA DID NOT TRACK WITH QMM DATA 

The author finds an inconclusive correlation between QMM and TPM/EVM. The 
data given in Table 3 for TPM/EVM did not track with the data in Tables 1 &2 for QMM 
survey questionnaires, even though software programs A & B provided data that might 
lead one to believe that there is a correlation between QMM and TPM/EVM. However, 
when it came to software program C the data proved to be inconclusive for a correlation 
to be present. The QMM survey questionnaires in program C reflected a very poor score 
of less than 70%, and their EVM/TPM score presented an acceptable score of 100% or 
one according to the requirements of large software programs. 

The author noted that the possible causes could be in the way the data is gathered, 
calculated and presented for TPM/EVM. This is based on the fact that TPM/EVM data is 

processed, calculated and presented during meetings that are held weekly by Project 

27 



Managers for project status. Then the weekly TPM/EVN data is summarized and 
presented for the monthly general staff presentation. This gives the Project Managers 
time so they can make adjustments each week reflecting an acceptable status for the 
monthly general staff presentation. Also, new requirements or changing requirements 
constantly being placed on a project make TPM/EVM goals difficult at best. 

The author noted that the QMM questionnaire surveys gives better details of 
where in the program the Program Managers are possibly having difficulties. QMM 
concentrates on four basic areas in the questionnaire surveys such as: requirements 
management, estimation/planning management, people management and risk 
management. The TPM/EVM data gives a yes or no answer to Program Managers on 
weekly and monthly status in meeting the projects technical goals. 

E. SUMMARY 

In this Section III the author presented the data from QMM questionnaire surveys 
to measure the perception of program managers from programs A, B and C that have the 
responsibility for the software development within each of the said programs for the U.S. 
Army. The author then gathered TPM/EVM data using a metric methodology from the 
same programs given the questionnaire surveys, and developed both sets of data tables 
for review of possible correlation between them. The author noted during his review of 
these data tables that the TPM/EVM data did not track with the QMM data presented. 
Therefore an inclusive correlation between QMM and TPM/EVM was presented. 

In Section IV the author presents Conclusions, Recommendations, and 
suggestions for Future Work based on his findings. 
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IV. CONCLUSIONS, RECOMMENDATIONS AND FUTURE 

WORK 


A. CONCLUSIONS 

This thesis provided an initial evaluation of QMM and TPM for a possible 
correlation between the two when evaluating software management and technical 
performance on specific software programs. The software programs evaluated varied 
considerably and played a significant part in the overall success of a larger software 
program. The decisions and policies that program managers make using QMM and 
EVM/TPM could provide the advantage given if there were a correlation between them. 
However, Earned Value Management (EVM) did not track with QMM as test data 
reflected. Also, the author notes that EVM/TPM did not indicate the program as 
successful or non-successful as QMM provided in test data reflected in section III 
showed an inconclusive correlation between QMM and TPM/EVM. 

1. QMM Survey 

The author used the survey format provided from Martin J, Machniak thesis, 
Development of a Quality Management Metric (QMM) Measuring Software Program 
Management Quality December 1999. The format of the QMM survey, and the individual 
questions and the TPM data was unchanged. The intent of this thesis was to find out if 
there is a correlation between QMM and TPM. This thesis achieved the goal by surveys 
and EVM data taken from three software programs found on a major Army software 
program. The surveys were done in an acceptable amount of time by the dedicated 
participants in programs A, B and C. The survey completion time was on average 
approximately 90 minutes. The time needed to take the survey ranged from 60 to 121 
minutes approximately. 

2. QMM Scores 

All three programs A, B, and C, having QMM percentage scores in comparison to 
the individual overall program success scores reflected a strong correlation between 
them. However, the author felt that the program managers in answering the survey 
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questions needed to be told not to answer the survey on how they think a program should 
be managed but how their program is actually run. All but one of the three programs had 
an overall success score of seven, which corresponded to the QMM percentage score of 
70 percent plus or minus 5 percent points, and indicated a strong correlation according to 
Martin J. Mackniak QMM standards defined in his thesis. QMM scores compared to the 
QMM Standard scores show a strong correlation. Two out of three survey participants 
recorded QMM percentage scores within 5 points of the mean success score for their 
respective programs. The only exception was program C where it fell below the QMM 
standard of 70 percent. However, this program was dealing with very immature critical 
technologies and constant change in requirements of which the program manager did not 
have control over such changes. 

3. TPM DATA 

The TPM data was taken for a ten month period using EVM which was reported 
monthly using the SPI and CPI data. The performance processes of the TPM/EVM 
should be measured, recorded, and scheduled on a regular basis for full effectiveness of 
the process. The TPM reports that are not reported or have been ignored can be 
considered proof that the TPM process is not being used and is a possible example of a 
non-valued added activity. However, if indicators for ignoring one particular TPM are 
justified then it should be closed out and no longer reported. The reportable SPI and the 
CPI of the TPM should go to the program manager and IPT keeping everyone fully up¬ 
dated. Three out of the three programs that participated recorded EVM percentage scores 
within the set standard. In the EVM the standard is set at a ratio of less than one which 
implies that work is underachieving against the plan, and above one implies better than 
planned, where 100 percent is equal to one This is the acceptable standard set for large 
software programs within the U.S. Army. 

4. TPM DATA DID NOT TRACK QMM DATA 

The author notes that in the case of program C which had constant changes in 
their TPM/EVM technical requirements gave an inconclusive correlation between the 
QMM data and TPM/EVM data in section III. 
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B. RECOMMENDATIONS 

The author feels that by using both the TPM (in the EVM format), and the QMM 
survey questionnaires as possible tools, software program management performance can 
be improved through complete evaluation of EVM data, and QMM survey questions data. 
The dichotomies found in the QMM questionnaire survey by participants in the same 
software program during the same time period need to be discussed. When the program 
manager notes a change in the EVM data where the TPM does not meet the goals set-up 
in the program, the QMM survey should be given and meetings will need to be called to 
discuss differences in survey questions, and change in TPM status. 

1. Evaluation of the Survey Sections 

As new changes present themselves to the software engineering field, new 
techniques, followed by different strategies become the norm. The need to re-evaluate the 
survey sections to reflect and refine the need for better software quality is a must. The 
Program Manager can read the survey questions having a view of the past and present 
performance of their software program, and then look for sections that score the lowest as 
possible areas for improvement. 

There is a need to have an administrator for the survey to help with the 
explanation to various levels of program management, plus to help uncover any 
misperceptions and possible pre-bias that can exist in giving survey results. 

Also, there is a need to focus on survey questionnaire development to reflect the 
continuous changes in software management and philosophy as an on-going process: a) 
Concept clarification with keeping current program condition as the object of each survey 
question; b) Survey question replacement to reflect new trends in quality software 
management; c) Giving upper program management the option to change the sectional 
point value of the questions in order to help determine software management quality; and 
d) Keeping the survey length and time to complete to a minimum. 
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2. 


Evaluation of TPM/EVM Sections 


Using EVM/TPM as a performance management tool can ensure a project is 
provided the best possible cost and schedule impact with the potential to offer 
organizations significant benefits around monitoring and managing software programs. 
However, the difficult part is to ensure that the EVM approach focuses squarely on the 
right software projects and monitors them at the right level, because EVM solutions have 
failed in the past by becoming too complicated, and therefore cannot be maintained by 
the organization. Having experienced EVM and TPM personnel on staff would help to 
establish an understanding of what a successful EVM solution would look like in their 
dynamic environment. Also, by reviewing the core topics such as: a) determining the 
strategic priorities of the software project; b) ensuring that the EVM analysis focuses on 
the right aspects of the projects; c) designing and building the EVM solution that suits 
your project needs and provides the appropriate level of detail; d) reporting EVM finding 
in a way that everyone can understand them; e) building EVM right into the software 
program budgeting process. 

3. TPM and QMM Metrics 

In this thesis the author provided an informal verification, validation, and 
evaluation of only three software programs for the QMM and the TPM/EVM scores. All 
three of these programs fell under the Department of the Army. The author felt that due 
to the nature in which software programs are managed in this environment and not in the 
civilian work place many more software programs of various sizes and variety need to be 
considered before establishing a statistically sound correlation between QMM score to 
overall software program success and TPM using EVM data. The metric formulation in 
scoring will require possibly different coefficients, and should be based on the software 
program size, complexity and environment, whether commercial or government. 

As tools for measurement improve and are developed and what is considered a 
quality software program is defined, improvements will continue to come forth whether 
as QMM or TPM. The author noted that the data between QMM and TPM/EVM, even 
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though it gave an inconclusive correlation in this thesis, may be developed further in the 
future as we search how to make a successful software program. 

4. Metric Scoring of QMM and TPM/EVM 

In this thesis informal verification and validation, provided evaluation for three 
software programs for a correlation between QMM and TPM/EVM of which all of the 
programs were Department of Defense systems. The author suggested a larger sample 
should be taken of software program managers and key team members, plus a greater 
variation of software programs need to be evaluated using QMM surveys and TPM/EVM 
data before a well defined correlation can be established between the QMM score and the 
TPM/EVM data with respect to overall success of a software program. The author noted 
that to establish a template to evaluate improvement in software management 
performance may involve repetition of a computational procedure. This replication of a 
cycle of operational procedure may result in an approximate desired outcome that is 
closer to the set goals. The formulation of coefficients for the QMM surveys and 
EVM/TPM data may need to vary according to top management’s overall program goals, 
for example, taking size, use and complexity of the software being developed into 
consideration. Software Program Managers must customize their approach with respect 
to any use of available measurement software tools such as QMM surveys and 
TMP/EVM data, keeping project goals as flexible as possible for directional changes by 
top management. 

C. FUTURE WORK 

In this section the author would like to make it clear to the reader that he did not 
address the relationship between Earned Value (EV) and QMM. This, the author has left 
as possible future work as stated within the sections of this thesis. 
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APPENDIX A 


A. PROGRAM A - PROGRAM MANAGER 
1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

47 

e 

49 

96 


0.92 

= 

88.32 

Est./Planning 

Management 

D 

39 

D 

59 

98 


0.67 

= 

65.66 

People 

Management 

c 

45 

g 

46 

91 


1.86 

= 

169.26 

Risk Management 

D 

55 

D 

46 

101 


0.55 

= 

55.55 


QMM SCORE 378.79 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 


QMM percentage score: 


77.35% 


Objective/Subjective view of the overall success of program A on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Program Manager 
Success Score: 8 
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2 , 


Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e_Yes No N/A 


1 

PM chose to have a formal requirements list 

X 



2 

Requirements recorded in some way 

X 



3 

Written requirements were part of some formal document 

X 



4 

Written requirements were informal 


X 


5 

At least some requirements were oral only 


X 


6 

All stakeholders were identified 


X 


7 

All stakeholders participated in the requirements extraction 

X 



8 

Some stakeholders participated in the requirements extraction 

X 



9 

Management extracted requirements, no stakeholder involvement 


X 


10 

Management passed requirements to development team 

X 



11 

Stakeholders not involvved in Management extraction, but approved 


X 


12 

Management gets inputs from stakeholders, then develops requirements 

X 



13 

Developers work informally with users to arrive at requirements 


X 


14 

Same as 13, but management oversees and formalizes 


X 


| If a waterfall or sequential development strategy: | 

15 

All requirements complete before design 


X 


16 

Some requirements left incomplete prior to design 

X 



17 

Requirements informal prior to design effort 


X 


18 

Requirements serve as input 

X 



19 

Length of time for requirements work greater than development work 


X 


20 

Requirements developed in parallel to design 

X 



| OR If a prototype, throwaway, or other development strategy: | 

15 

Learn about requirements through development efforts 




16 

No coding until all requirements are defined 




17 

Requirements formal prior to design effort 




18 

Requirements serve as output 




19 

Requirements definition work in parallel to development efforts 




20 

Requirements developed in parallel to design 




21 

Are requirements frozen at some phase 

X 



22 

Change management exists 

X 



23 

Change management is formal 

X 



24 

Project strategy is consistent throughout development 

X 



25 

Requirements are updated 

X 



26 

Configuration Management (CM) exists 

X 



27 

CM is formal 

X 



28 

Requirements are testable 

X 



29 

Requirements testing considered/implemented during extraction 

X 



30 

Requirements testing plan exists 

X 



31 

Requirements testing is formal 

X 



32 

All requirements have priorities 


X 


33 

All requirements must be implemented 

X 



34 

Requirements are tested 

X 



35 

All requirements are equally important 

X 



36 

At least some requirements have priorities 


X 


37 

All requirements are traceable 

X 



38 

Traceability not important 


X 


39 

Each requirement has an author 

X 



40 

Who authored requirement is not important 

X 



41 

Initial set of requirements to be implemented, no requirements creep 


X 


42 

Structured and tracked changes to requirements only 


X 


43 

Change is inevitable, changes allowed at all times 


X 


44 

Change is inevitable, but changes limited 

X 



45 

Requirements control funding 

X 



46 

Requirements history kept 

X 



47 

Baseline established for requirements at some point prior to develop 

X 



TOTAL SCORING 

EM 

IB 
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Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f_Yes No N/A 


1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 


X 


4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

X 



5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 


X 


7 

Time measure process metric used (cycle time) 


X 


8 

Process metrics tracked and updated throughout program execution 


X 


9 

Program cost estimations made from product or process metrics 

X 



10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 


X 


12 

Program's primary purpose, including major functions and deliverables known 

X 



13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 

X 



15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 


X 


18 

Quality assurance plan or similar to aid in detecting defects early in program 

X 



19 

COCOMO estimates performed 

X 



20 

CSCI clearly defined and tasked 

X 



21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 


X 


29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 

X 



33 

Automated program tracking used 


X 


34 

PM usually thorough in tracking and reporting schedules and financials 

X 



35 

WBS developed only as data call 


X 


36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 

X 



38 

Estimations are done using both top down and bottoms up approaches 

X 



39 

All program team members involved in planning process 

X 



40 

Hardware also considered in estimaation process 

X 



41 

Program history compiled 


X 


42 

System upgrades (SCR) software changes requests estimated individually 

X 



43 

Management duties apart of each team member's responsibilities 


X 


44 

PM dictates schedules to program team 


X 


45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 

X 



47 

Test planning done at the start of the program 

X 



48 

Estimations are completed by those performing the tasks 

X 



49 

Sensitivity analysis performed for program choices 


X 


50 

Software deployment planning completed 

X 



TOTAL SCORING 

lea 

B3 
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People Management Questionnaire Responses 


No. People Management Questionnaire - Total Block g_Yes No N/A 


1 

PM is accessible in person by each team member 


X 


2 

PM is accessible via email (memo, letter) by each team member 

X 



3 

PM is accessible via phone by each team member 

X 



4 

PM not only considers a person's suitability, not also desire to be on a team 

X 



5 

PM consults with each team member regarding their career goals 

X 



6 

PM regularly holds meetings to inform team of program progress 

X 



7 

PM solicits opinions from team members before making decisions 

X 



8 

PM lets teams make decisions affecting their work 


X 


9 

PM freuently makes decisions without any consultation with members 


X 


10 

PM understands the technology/language of the program 

X 



11 

PM is able to communicate with other the technical issues in the program 

X 



12 

PM prioritized problems or conflicts within the program 

X 



13 

PM assists team members in developing/advising of career path 




14 

PM empowers program members to recommend hiring new team members 

X 



15 

PM empowers program members to recommend firings of other members 


X 


16 

PM specifically assigns work to each program member 


X 


17 

PM sets communication protocol 

X 



18 

PM allows unrestricted communications 

X 



19 

PM encourages or requires training for each individual 

X 



20 

PM takes control in difficult/roblem areas 

X 



21 

PM looks ahead to new programs, new upgrades of existing program 

X 



22 

PM maintains regular communications with all stakeholders 

X 



23 

PM maintains regular communications with users 


X 


24 

PM encourages program team communication with users 

X 



25 

PM encourages program team communication with stakeholders 

X 



26 

PM facilitates horizontal communication within program 

X 



27 

PM facilitates communication during integration 

X 



28 

PM holds meetings without clear objectives 


X 


29 

PM must approve all decisions within the program 


X 


30 

PM must approve ail interactions with stakeholders 


X 


31 

PM must approve all interactions with users 


X 


32 

PM makes all presentations to stakeholders/users 


X 


33 

PM is considered "flexible" in terms of program members personal issues 

X 



34 

PM, at least occasionally, schedules/promotes outside work team activities 

X 



35 

PM is readily willing to listen to program prblems and complaints 

X 



36 

PM takes action to resolve program problems and complaints 

X 



37 

PM is generally respected by stakeholders, users, and organization 

X 



38 

PM sometimes fails to grasp important technical issues in program 



X 

39 

PM recruits program team members from outside organization 



X 

40 

PM participates in technical reviews 

X 



41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual's tasks are specific, each exposed to the "bigger picture" 

X 



43 

PM has clearly defined his/her expectations for each individual 

X 



44 

PM delegation of duties is usually seemless in execution 

X 



45 

PM acts as facilitator to solving personnel conflicts 

X 



46 

PM attempts to motivate individuals on the program team 

X 



47 

PM clearly spearates technical from managerial roles for individuals 


X 


48 

PM directs how he/she expects the task to be accomplished 


X 


49 

PM directs what needs to be done, but does not direct how 

X 



50 

PM attempts to spotlight individuals in the program for positive exposure 

X 



TOTAL SCORING 

EH 

m 
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Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Block h_Yes No N/A 


1 

Risk Management (RM) is specifically an activity in the program 


X 


2 

RM is formal and documented 


X 


3 

A specific RM Ian exists 


X 


4 

RM is required in the program, but not used during the program 


X 


5 

RM is done prior to the program execution 


X 


6 

RM is done by an outside entity to the development 


X 


7 

RM is done internally only 

X 



8 

RM is both internally performed and externally assessed 


X 


9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is only a management function 

X 



11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 


X 


13 

Risks are only generalized 

X 



14 

Each risk is delineated 


X 


15 

Each risk has a consequence 


X 


16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 


X 


18 

Risk Management is automated 


X 


19 

Risks are tracked 


X 


20 



X 


21 

Regret analysis performed 


X 


22 

RM drives decisions in the program 


X 


23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 

X 



25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 


X 


28 

Risk Assessment done prior to program start 


X 


29 

Risk Assessment includes personnal risk 

X 



30 

RM uses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 

X 



34 

Risk Assessment is briefed organization structure above program manager 


X 


35 

Risk Assessment includes requirements risks 

X 



36 

Risk Assessment includes user risks (too little involvement of user) 

X 



37 

Risk Assessment includes documentation risks 

X 



38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 

X 



42 

Documentation proof exists to demonstrate following risk management plan 

X 



43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 

X 



45 

Other organizational checklists used for risk assessment 


X 


46 

Internal organizational checklists used for risk assessment 

X 



47 

Risk Assessment information contributed to internal or other database 


X 


48 

Risk Assessment includes internal organization risks 

X 



49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 

m 

Eel 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

X 

informal requirement list 


written requirements 

X 

oral requirements 


requirements informal, but recorded 


requirements not recorded 

X 

requirements as part of an SRS (or other formal repository) 


requirements informally recorded 

X 

requirements taken as is from customer 


look to reformulate, interview in-depth, or otherwise re-validate 

X 

only one development strategy used 


strategies not consistent, used at different times 

X 

stakeholders as part of requirements development 


stakeholders approving requirements after formulated by development team 

X 

requirements are testable 


requirements have no test plans 

X 

informal test plan or no test plan 


formal test plan 

X 

test team involved with requirements 

X 

no test team input or plans during requirements development 


only a percentage of requirements present in baseline 


baseline must contain all requirements 

X 

requirements documentation has hierarchical structure 


all requirements must be implemented 

X 

requirements have listed responsible party 

X 

requirements origin not important 


requirements documentation have versions 

X 

no requirements history 


requirements have specific attribute values 

X 

requirements all rank evenly 


funding controls requirements definition 


requirements definition controls funding 

X 

reqquirements are top down 

X 

requirements are bottom up 


users/stakeholders are identified and interviewed (market survey) 


no special consideration to identify users/stakeholders 

X 

each requirement has a singular concept 


some requirements are compound statements 

X 

requirements definition minimized when funding short 


program scope may reduce, but requirements definition completed 

X 

requirements extraction has formal process 

X 

requirements extraction ad hoc 


change procedures formal 

X 

change procedures ad hoc 


users/stakeholders somehow involved in requirements definition 

X 

program team only involved in requirement definition 


management sets requirements for developers 


developers at least partially involved in setting requirements 

X 

requirements changed at least once since baseline established prior to new version 

X 

requirements in baseline has not changed prior to new version or upgrade 


no ranking of requirements 


requirements have priorities assigned 

X 

use-case diagrams (or other models or scenario developments) 

X 

no models used for requirements extraction 


requirements changes informal 


requirements changes formal 

X 

plan to "freeze" requirements at some designated milestone 

X 

no provision for "freezing" requirements 


requirements must be traceable 

X 

origin of requirements not important 


requirements must be testable 

X 

system developed must be testable 


test plans to determine requirements implemented 

X 

no test plans needed for requirements verification 


requirements have priorities in implementation 


all requirements must be implemented 

X 

some requirements have multiple statements or ideas 

X 

one idea, one statement per requirement 



Requirements Management (page 1 of 2) score 


Hsl 


40 











































































































Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) 

requirements first, then initial development work 


initial development work then requirements 

X 

requirements documentation driving development 


requirements documentation developed in parallel/after development 

X 

user feedback considered during development 


after development starts, user feedback serves as input to new work 

X 

change management procedures used strictly 

X 

change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 

X 

design decisions only after approved requirements stabilized 


requirements summarized wht we have developed 


requirements are the blueprint for development 

X 

length of time for requirements work greater than development work 

X 

length of time for requirements work less than development work 

X 

requirements have design detail 


no design detail in requirements 

X 

requirements creep to be avoided 


requirements creep o.k., but need to be controlled 

X 

freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 

X 

informal change procedure 


change management plan 

X 

no change management plan 


requirements ambiguity always present to some extent 

X 

requirements ambuiguity unacceptable at any level 


testing considered up frornt during requirements determination 


testing considered down the line during development 

X 

requirements development team members different from implementation 


those working on requirements, work on implementation 

X 

start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 

X 

ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWAY, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 

develop prototype, then determine requirements 


determine requirements prior to any development work 


requirements testing done after each iteration 


no testing 


individual changes as necessary 


only block changes made 


development team decides on changes after iteration 


users involved with changes 


changes based on feedback only from user for correction of problems 


changes to upgrade system and correct problems 


funding controls changes and change procedures 


changes control funding 


requirements documentation finalized prior to development 


requirements fluid throughout development (only freeze at end) 


requirements test plans completed prior to development 


requirements test plans completed after development 


requirements first, then initial development work 


initial development work then requirements 


use development effort to learn more about requirements 


define all requirements prior to coding anything 


requirements ambiguity always present to some extent 


requirements ambiguity unacceptable at any level 


requirements have design detail 


no design detail in requirements 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


get something to users as soon as possible for evaluation 


make sure it is complete before releasing 


management dictates requirements 


development team visually represent requirements through rapid prototyping 


new requirements allowed after initial requirements defined 


new requirements not allowed 



Requirements Management (pg 2 of 2) score [12] +pg 1 score [35] = TOTAL SCORE [47] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 


formal derivation of product metric for estimation of size 

X 

ad hoc size estimation 


ad hoc process evaluation 


formal derivation of at lest one process metric 

X 

develop work breakdown structure (WBS) 

X 

assign work as needs arise 


estimates are developed to fulfill a data call only 


use estimates to plan program 

X 

use estimates to sell program only 


estimates are useful to the project tema for planning purposes 

X 

resource evaluations made for program 


no resource evaluation for planning 


use both bottom up & top down for estimate, use one stakeholders like 

X 

use both bottom up & top down and evaluate significant differences 


estimates made and not updated 

X 

estimates updated throughout program 


resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 


estimations made to fit budget 


budget made from estimations 

X 

estimations compromised to get program 


rather risk loss of program than compromise confident estimations 

X 

cycle time estimations 


no cycle time estimations 

X 

event count estimations 


no event count estimations 

X 

lines of code (LOC) estimation 

X 

no LOC estimation 


function pont (FP) estimation 

X 

no FP estimation 


estimates by algorithmic methods 


estimates by analogy 

X 

expert judgement for estimates 

X 

ad hoc estimates 


estimates by algorithmic methods 

X 

ad hoc estimates 


expert judgement for estimates 

X 

estimates by analogy 


ad hoc estimates 


estimates by analogy 

X 

bottom up estimates 

X 

expert judgement 


top down estimates 

X 

expert judgement 


ad hoc estimates 


any other estimate process 

X 

fuzzy logic estimating method 

X 

no formal estimation methodology 


WBS development from estimates 

X 

WBS development in parallel or prior to estimation completion 


critical path of program determined 


tasks developed but no path is identified 

X 

estimators are program team members 

X 

estimators are outside program team 


management only on estimations 


all team members involved in estimation process 

X 

estimates updated at reviews 


no updates of estimates 

X 

estimates updated at reviews 

X 

estimates constantly updates (in between reviews, to) 


estimate procedures stay the same 

X 

estimate procedures change 


stakeholders are part of estimation process 


stakeholders brief estimations after completion 

X 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 


WBS has objective measure of completeness 


important to have WBS as guide, not rigid implementation 

X 


Estimation/Planning Management (page 1 of 2) score 


25 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 


estimates for program initiation only 

X 

system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as whole 


estimates for on-gong resources needed to maintain s/w 


estimates for maintenance not done 

X 

informal re-estimates during development 

X 

formal re-estimates at pre-defined milestones 


formal re-estimates when amendment changing the system is introduced 

X 

informal re-estimates when amendment changing the system 


person in-charge of estimation walks in a managers office to get an opinion 

X 

meeting(s) organized for purpose of performing cost estimations 


factor analysis prior to commencement of program 


none done 

X 

change control procedures set in place 

X 

no set procedures 


elapsed time and actual work time estimates 


one or the other or neither 

X 

no schedule created 

X 

scheudle created 


schedule not updated 

X 

schedule updated 


schedule followed 

X 

schedule not followed 


tasks identification arises as program progresses 

X 

detailed level tasks identified prior to program initiation 


scope of program understood by all 


scope not explicitly defined 

X 

quality factors and criteria identified 

X 

no explicit quality factors defined 


no project tracking tools used 


project tracking tools used 

X 

CSCIs identified and tasked 

X 

CSCIs not explicitly identified 


expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 


no cost schedule developed 


cost schedule developed 


no resource schedule developed 

X 

resource schedule developed 


team members, management know at any time if in budget & schedule 


exact budget & schedule status somewhat unclear to at least some 

X 

individual program phases are estimated 

X 

only top level program estimated 


stakeholders/users emphasis understood-quick to field or all complete 


program management sets delivery tradeoffs without outside input 


testing planned with initial program planning 

X 

testing not in initial planning 


documentation not considered ininitial planning 

X 

documentation part of initial planning 


hardware considered in estimations 

X 

software only considered 


no formal schedule/cost tracking 

X 

formal procedures established for tracking cost and schedule 


earned value set up 

X 

earned value not used 


estimations omit documentation planning 

X 

documentation in estimates 


training omitted in estimates 

X 

training part of estimates 


earned value set up, but not tracked 


earned value tracked 

X 

detailed planning done with incomplete set of requirements 

X 

detailed planning done with detailed set of requirements 


complete infrastructure support mechanism understood for estimations 


no consideration of infrastructure done for estimations 

X 

team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 


work breakdown structure (WBS) set up 


no WBS completed 

X 

Estimation/Planning Management (pg 2 of 2) score [14] +pg 1 score 

[25] = 

TOTAL SCORE [39] Enter on QMM scoresheet blk b. 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 1 of 2): 
Human Resources 


program team members have clearly deined, segmented roles 

X 

work responsibilities are shared 


formal team building procedures are used 


no formal team building emphasized 

X 

program manager flexible regarding work hours 

X 

program manager maintains strict standards for work hours 


big picture conveyed to all team members by program management 

X 

program management focuses on the partitioned tasks with team 


people issues dealt with primarily through indirect methods (email, memo, etc) 


people issues dealt with primarily through direct methods (face-to-face) 

X 

training is required and planned on a regular basis 


training is ad hoc 

X 

each team member is educated on and understands overall program and their roles 


team members only know their respective areas 

X 

consideration for team members' career goals are reflected in assignments 

X 

team members must adapt to tasks that are assigned 


team members assignments and responsibilities are mostly dictated by PM 


assignments and responsibilities are discussed and agreed upon with PM 

X 

management leads in problem solving 


management facilitates and lets team lead in problem solving 

X 

management welcomes problems as challenges and opportunities 

X 

management views problems as obstacles and grounds for punishment 


team members participate in performance evaluations of peers 


Personnel evaluations are strictly PM responsibility 

X 

management reinforcement feedback sparse and inconsistent, if any 

X 

management provides timely reinforcement feedback for positive behaviors 


management provides basic needs of office facilities fairly well 

X 

office facilities are a drawback to working in the program 


working conditions are fairly comfortable, time off policy fairly good 

X 

working conditions and time off policy is inconsistent and difficult at times 



Communication: 


communications primarily written (email) 


communications primarily verbal (face-to-face) 

X 

detailed instructions: oral presentation, follow-up email 

X 

email only 


formal communication protocol 


informal communications 

X 

external vertical communications restricted 

X 

external vertical communication allowed 


coders notebook weekly accomplishment reports required 

X 

not required 


user-coder relationship established, encouraged, and mediated 


user-coder interaction minimized 

X 

meetings structured to minimize waster time 

X 

meetings unstructured and open ended 


meetings have agenda, objectives, and conclude with action items 

X 

meeting agenda fluid and open ended 


program management and coder communication face to face 

X 

program management and coder communication primarily email 


program team updated regularly regarding organizational & program status 


meetings infrequently scheduled 

X 

open communications is encouraged 

X 

communication hrough chain of command only is encouraged 


program manager accessible for discussions 

X 

program manager difficult to get an appointment to see 


program management (PM) is viewed as separate from team 


PM mixes with team frequently 

X 

management regularly holds team meetings 

X 

meetings are sporadic 


meetings are structured with definite goals and objectives 

X 

meetings are informal 


program management is generally easy to reach and talk to 

X 

PM is usually hard to get a hold of and difficult to talk to 


team-program manager relationship adult-adult 

X 

team-program management relationship parent-child 


schedules are spontaneous and poorly communicated 


schedules must be fixed and rigidly followed and formally reported 

X 

work is seen as complex processes involving team working together 

X 

work broken into pieces with minimal team member interaction 


action items often is poorly disseminated and usually not followed through 


action items communicated and followed through thoroughly 

X 

team members require frequent clarifications by PM for assigned tasks 


team members rarly require clarifications by PM for assigned tasks 

X 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 


short tern program and immediate w ork focus 

X 

lead through personal attention to others 


action-oriented leadership approach 

X 

run as much of the organization as possible 


let team make decisions as much as possible 

X 

direct and domineering style 

X 

encourage independence in others 


traditional leaders respect hierarchy 


do w hat needs to be done 

X 

w in cooperation rather than demand it 

X 

tough-minded with others 


act strongly and forcefully in the field of ideas 

X 

prefer to lead other independent types while seeking auto no my for self 


consults w ith team members to find solutions to problems 

X 

consults team members to get validation of PMs predetermined solutions 


keep people w ell informed 

X 

only as much knowledge as necessary for their work 


make things happen by focusing on the immediate problems 

X 

long range focus and de-emphasize current problems 


manage others loosely and prefer minimal supervision 


follow traditional procedures and rules conscientiously 

X 

leadership, management decisions exclusively by program management 


program management makes decisions but gets inputs from team 

X 

team-program manager relationship adult-adult 

X 

team-program management relationship parent-child 


program management makes decisions but gets inputs from team 

X 

all program team members responsible for program decisions 


w hen a problem arises: management takes over to solve it 

X 

management lets the team solve the problems 


leadership is do as 1 say, not do as 1 do 

X 

leadership by example 


program expectation not influenced by PM 


program expectation managed by PM 

X 

PM gives freedom to team, but has no mentoring for members (abdication) 

X 

PM empow ers teams by mentoring members to be leaders 


promgram management w aits and sees w hat happens then plans 


management plans far in advance 

X 

program management is constantly reacting to emergencies 

X 

management is one step ahead of problems 


facilitative approach to solving problems 


take charge readily and often 

X 

program management is complex, takes much time to understand 

X 

management is simple, easy to figure out 


program management prefers to plunge right in 

X 

takes time to separate things to be done and order of doing them 


program management reacts spur of the moment 

X 

methodically follow s plans 



Technical Competency of the Program Manager: 


PM has technical experience particular to the particular s/w program 

X 

PM relies on team members solely 


PM participates in technical reviews 

X 

PM only in non-technical reviews 


PM participates in making technical decisions when problems arise 

X 

PM delegates technical questions 


PM does not get involved discussing technical options 


PM contributes to technical options being discussed 

X 

PM does not review technical options and decisions 


PM reviews technical options and decisions 

X 

PM actively attempts to keep up-to-date with current techno logy and standards 


PM is removed from cutting edge technology issues 

X 

PM receives technical periodicals and occasionally references applicable articles 

X 

PM doesn't read periodicals nor reference current articles to team 


PM doesn't have technical background (or education) 


PM has technical background (or education) 

X 

team members avoid PM w hen they need technical advice 


team members generally consider talking to PM regarding technical issues 

X 


HR[9] + Comm. [17] + Leadership [11] + Tech. Competency [8] = People Mgmt. score [45] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

X 

RM is informal, if at all 


a risk management plan exists 

X 

no risk management plan is developed 


RM is more of a data call than a useful document 


RM drives decisions on the program 

X 

RM is done prior to the program beginning 


RM is done prior and during program execution 

X 

RM is only done during the program execution 


RM is done prior and during program execution 

X 

risks are generalized through the whole program 


risks are categorized 

X 

risk management is done internally, only 


an outside organization also contributes to the RM process 

X 

risk is a management function 


risk is a program team function 

X 

risks are precisely articulated 


risks are generalized, if at all 


each risk has a consequence 

X 

consequences are generalized, if at all 


a mitigation strategy is completed for each risk 

X 

mitigation strategy is generalized, if at all 


contingency plans are developed for a RM plan 

X 

contingency plans are ad hoc as problems arise in the program 


risks are anticipated 


if problems arise, management will deal with it 

X 

the program doesn't have any risk 


programs that do not have risk, have problems 

X 

risk management is automated 


risk management may use tools, but depend on human input 

X 

risks are assigned probabilities 


probabilities are not relevant for RM 

X 

all risks are potential problems, relative priorities for risks are not useful 


risks are weighed relative to other program risks and thus prioritized 

X 

risk management information is only shared internally 


risk management information is shared with all stakeholders 

X 

risk analysis uses ordinal rankings 


risk analysis uses actual measurements with a mathematical model 

X 

regret analysis used 

X 

no regret analysis done 


attach probabilities to future events 

X 

no probabilities associated with future events 


assessing risks with mechanical meethods 


risks should be compared to other risks and sorted 

X 

risk status tracked 

X 

not tracked 


technical risks examined 

X 

no technical risks examined 


process risks examined 

X 

no process risks examined 


product risks examined 

X 

no product risks examined 


stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 


checklists used to identify risks 


no checklists used 

X 

risks are tracked 


no tracking or monitoring of risks 

X 

each risk has an impact 

X 

no impact analysis of risk 


each risk has a mitigation plan 

X 

no individual risk mitigation 


risks monitored by priority 

X 

no special attention to track higher priority risks 

X 

risk assessment is formalized 

X 

no formal risk assessment 


risk control is formalized 

X 

no formal risk control 


integration risks not considered 


integration risks examined 

X 


Risk Management (page 1 of 2) score 


I 30 I 


46 














































































































Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 


unforeseen risks have occurred in program 

X 

any risk that came up had been identified previously 


personnel risks examined 

X 

no personnel risks examined 


estimation risks examined 

X 

no estimation risks examined 


planning risks examined 

X 

no planning risks examined 


requirements risks examined 

X 

no requirements risks examined 


resource risks examined 

X 

no resource risks examined 


risk management plan updated regularly 

X 

no regular risk management plan updates 


risks charted 

X 

risks not charted 


performance risks examined 

X 

performance risks not examined 


program management self risks examined 

X 

no program management risks examined 


risk from program constraints examined 

X 

no program constraint risks examined 


each category of risks are prioritized 

X 

no prioritization 


each category of risks are evaluated for impact 

X 

no impact analysis performed 


each category of risks have control strategy 

X 

no control strategy 


documentation risks examined 

X 

no documentation risks examined 


regret matrix tracked 


no regret matrix or not tracked 

X 

communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 

X 

taxonomy-based questionnaire used to identify risks 


taxonomy-based questionnaire not used 

X 

associated hardware risks examined 

X 

no consideration for hardware risks 


integration risks examined 

X 

integration risks not examined 


communication risks examined 

X 

communication risks not examined 


leadership risks examined 

X 

leadership risks not considered 


risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 


risk documentation forms used 


no risk documentation forms used 

X 

dependency risks examined 

X 

no dependency risks examined 


alternatives like risk avoidance considered for high risk items 

X 

no consideration of risk avoidance 


documented risk statements use a condition-consequence type format 


condition-consequence of risk statements not clearly defined 

X 

no assignment of ownership of risk mitigation action 

X 

each risk mitigation action is assigned to an individual for resolution 


calculation of risk exposure made (probability X loss, for each risk) 


no risk exposure calculations 

X 

oral communication of risks only 

X 

risks written in a way that communicates nature and status of factors 


triggers used to quantify risk conditions present 

X 

risk conditions present are all subjective 

X 

risk "czar" in program for monitoring risks 


no special positions/responsibilities for risk monitoring 

X 

post-program review completed (scheduled) for unanticipated problems ID 


no post-program reviews completed or scheduled 

X 

no schedule risks examined 


risks to schedule investigated 

X 


Risk Management (pg 2 of 2) score [25] +pg 1 score [30] = TOTAL SCORE [55] Enter on QMM scoresheet blk d. 
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B. PROGRAM A - ASSOCIATE 

1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

44 

e 

51 

95 


0.92 

= 

87.4 

Est./Planning 

Management 

D 

52 

D 

47 

99 


0.67 

= 

66.33 

People 

Management 

c 

54 

g 

45 

99 


1.86 

= 

184.14 

Risk Management 

D 

55 

D 

43 

98 


0.55 

= 

53.9 


QMM SCORE 391.77 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 

QMM percentage score: 79.32% 


Objective/Subjective view of the overall success of program A on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Associate 

Success Score: 8 
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Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e Yes No N/A 


1 

PM chose to have a formal requirements list 

X 



2 

Requirements recorded in some w ay 


X 


3 

Written requirements were part of some formal document 

IT 



4 

Written requirements were informal 

IT 



5 

At least some requirements were oral only 


~ir 


6 

All stakeholders were identified 

IT 



7 

All stakeholders participated in the requirements extraction 

IT 



8 

Some stakeholders participated in the requirements extraction 

IT 



9 

Management extracted requirements, no stakeholder involvement 


~ 


10 

Management passed requirements to development team 

IT 



11 

Stakeholders not involvved in Management extraction, but approved 




12 

Management gets inputs from stakeholders, then develops requirements 

IT 



13 

Developers work informally w ith users to arrive at requirements 

IT 



14 

Same as 13, but management oversees and formalizes 

IT 



If a waterfall or sequential development strategy: | 

15 

All requirements complete before design 


X 


16 

Some requirements left incomplete prior to design 




17 

Requirements informal prior to design effort 


IT 


18 

Requirements serve as input 

X 



19 

Length of time for requirements w ork greater than development w ork 

IT 



20 

Requirements developed in parallel to design 


IT 


OR If a prototype, throwaway, or other development strategy: [ 

15 

Learn about requirements through development efforts 

X 



16 

No coding until all requirements are defined 




17 

Requirements formal prior to design effort 




18 

Requirements serve as output 




19 

Requirements definition work in parallel to development efforts 

X 



20 

Requirements developed in parallel to design 

X 



21 

Are requirements frozen at some phase 

X 



22 

Change management exists 

X 



23 

Change management is formal 

X 



24 

Project strategy is consistent throughout development 

X 



25 

Requirements are updated 

X 



26 

Configuration Management (CM) exists 

X 



27 

CM is formal 

X 



28 

Requirements are testable 

X 



29 

Requirements testing considered/implemented during extraction 

X 



30 

Requirements testing plan exists 

X 



31 

Requirements testing is formal 

X 



32 

All requirements have priorities 

X 



33 

All requirements must be implemented 

X 



34 

Requirements are tested 

X 



35 

All requirements are equally important 


X 


36 

At least some requirements have priorities 

X 



37 

All requirements are traceable 

X 



38 

Traceability not important 


~ 


39 

Each requirement has an author 

X 



40 

Who authored requirement is not important 

X 



41 

Initial set of requirements to be implemented, no requirements creep 

X 



42 

Structured and tracked changes to requirements only 


ir 


43 

Change is inevitable, changes allowed at all times 


IT 


44 

Change is inevitable, but changes limited 

X 



45 

Requirements control funding 


~ir 


46 

Requirements history kept 

X 



47 

Baseline established for requirements at some point prior to develop 

X 



TOTAL SCORING 

EH 

~2~ 
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Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f Yes No N/A 


1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 


X 


4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

X 



5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 

X 



7 

Time measure process metric used (cycle time) 


X 


8 

Process metrics tracked and updated throughout program execution 

X 



9 

Program cost estimations made from product or process metrics 

X 



10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 


X 


12 

Program's primary purpose, including major functions and deliverables known 

X 



13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 


X 


15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 


X 


18 

Quality assurance plan or similar to aid in detecting defects early in program 

X 



19 

COCOMO estimates performed 

X 



20 

CSCI clearly defined and tasked 

X 



21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 

X 



29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 

X 



33 

Automated program tracking used 

X 



34 

PM usually thorough in tracking and reporting schedules and financials 

X 



35 

WBS developed only as data call 


X 


36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 

X 



38 

Estimations are done using both top down and bottoms up approaches 

X 



39 

All program team members involved in planning process 

X 



40 

Plardware also considered in estimaation process 

X 



41 

Program history compiled 

X 



42 

System upgrades (SCR) software changes requests estimated individually 

X 



43 

Management duties apart of each team member's responsibilities 


X 


44 

PM dictates schedules to program team 


X 


45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 

X 



47 

Test planning done at the start of the program 


X 


48 

Estimations are completed by those performing the tasks 

X 



49 

Sensitivity analysis performed for program choices 


X 


50 

Software deployment planning completed 

X 



TOTAL SCORING 

Em 

El 
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People Management Questionnaire Responses 


No. People Management Questionnaire - Total: Block g Yes No N/A 


1 

PM is accessible in person by each team member 


X 


2 

PM is accessible via email (memo, letter) by each team member 

X 



3 

PM is accessible via phone by each team member 

X 



4 

PM not only considers a person's suitability, not also desire to be on a team 

X 



5 

PM consults with each team member regarding their career goals 


X 


6 

PM regularly holds meetings to inform team of program progress 

X 



7 

PM solicits opinions from team members before making decisions 

X 



8 

PM lets teams make decisions affecting their work 

X 



9 

PM freuently makes decisions without any consultation with members 


X 


10 

PM understands the technology/language of the program 

X 



11 

PM is able to communicate with other the technical issues in the program 

X 



12 

PM prioritized problems or conflicts within the program 

X 



13 

PM assists team members in developing/advising of career path 

X 



14 

PM empowers program members to recommend hiring new team members 

X 



15 

PM empowers program members to recommend firings of other members 

X 



16 

PM specifically assigns work to each program member 


X 


17 

PM sets communication protocol 

X 



18 

PM allows unrestricted communications 


X 


19 

PM encourages or requires training for each individual 

X 



20 

PM takes control in difficult/roblem areas 

X 



21 

PM looks ahead to new programs, new upgrades of existing program 

X 



22 

PM maintains regular communications with all stakeholders 

X 



23 

PM maintains regular communications with users 

X 



24 

PM encourages program team communication with users 

X 



25 

PM encourages program team communication with stakeholders 

X 



26 

PM facilitates horizontal communication within program 

X 



27 

PM facilitates communication during integration 

X 



28 

PM holds meetings without clear objectives 


X 


29 

PM must approve all decisions within the program 


X 


30 

PM must approve all interactions with stakeholders 


X 


31 

PM must approve all interactions with users 


X 


32 

PM makes all presentations to stakeholders/users 

X 



33 

PM is considered "flexible" in terms of program members personal issues 


X 


34 

PM, at least occasionally, schedules/promotes outside work team activities 

X 



35 

PM is readily willing to listen to program prblems and complaints 

X 



36 

PM takes action to resolve program problems and complaints 

X 



37 

PM is generally respected by stakeholders, users, and organization 

X 



38 

PM sometimes fails to grasp important technical issues in program 

X 



39 

PM recruits program team members from outside organization 

X 



40 

PM participates in technical reviews 

X 



41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual's tasks are specific, each exposed to the "bigger picture" 

X 



43 

PM has clearly defined his/her expectations for each individual 

X 



44 

PM delegation of duties is usually seemless in execution 


X 


45 

PM acts as facilitator to solving personnel conflicts 


X 


46 

PM attempts to motivate individuals on the program team 

X 



47 

PM clearly spearates technical from managerial roles for individuals 

X 



48 

PM directs how he/she expects the task to be accomplished 


X 


49 

PM directs what needs to be done, but does not direct how 

X 



50 

PM attempts to spotlight individuals in the program for positive exposure 

X 



TOTAL SCORING 

El 

B 
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5, 


Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Total: Block h Yes No N/A 


1 

Risk Management (RM) is specifically an activity in the program 

X 



2 

RM is formal and documented 

X 



3 

A specific RM Ian exists 


X 


4 

RM is required in the program, but not used during the program 


X 


5 

RM is done prior to the program execution 

X 



6 

RM is done by an outside entity to the development 


X 


7 

RM is done internally only 


X 


8 

RM is both internally performed and externally assessed 


X 


9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is only a management function 


X 


11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 


X 


13 

Risks are only generalized 


X 


14 

Each risk is delineated 


X 


15 

Each risk has a consequence 

X 



16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 


X 


18 

Risk Management is automated 

X 



19 

Risks are tracked 

X 



20 





21 

Regret analysis performed 

X 



22 

RM drives decisions in the program 

X 



23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 


X 


25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 

X 



28 

Risk Assessment done prior to program start 

X 



29 

Risk Assessment includes personnal risk 

X 



30 

RM uses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 

X 



34 

Risk Assessment is briefed organization structure above program manager 

X 



35 

Risk Assessment includes requirements risks 

X 



36 

Risk Assessment includes user risks (too little involvement of user) 


X 


37 

Risk Assessment includes documentation risks 

X 



38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 

X 



42 

Documentation proof exists to demonstrate following risk management plan 


X 


43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 


X 


45 

Other organizational checklists used for risk assessment 


X 


46 

Internal organizational checklists used for risk assessment 

X 



47 

Risk Assessment information contributed to internal or other database 


X 


48 

Risk Assessment includes internal organization risks 

X 



49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 


0 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

X 

informal requirement list 

w ritten requirements 

X 

oral requirements 

requirements informal, but recorded 

X 

requirements not recorded 

requirements as part of an SRS (or other formal repository) 

X 

requirements informally recorded 

requirements taken as is from customer 

X 

look to reformulate, interview in-depth, or otherwise re-validate 

only one development strategy used 


strategies not consistent, used at different times 

stakeholders as part of requirements development 


stakeholders approving requirements after formulated by development team 

requirements are testable 

X 

requirements have no test plans 

informal test plan or no test plan 


formal test plan 

test team involved w ith requirements 

X 

no test team input or plans during requirements development 

only a percentage of requirements present in baseline 


baseline must contain all requirements 

requirements documentation has hierarchical structure 


all requirements must be implemented 

requirements have listed responsible party 


requirements origin not important 

requirements documentation have versions 

X 

no requirements history 

requirements have specific attribute values 


requirements all rank evenly 

funding controls requirements definition 

X 

requirements definition controls funding 

reqquirements are top dow n 

X 

requirements are bottom up 

users/stakeholders are identified and interviewed (market survey) 

X 

no special consideration to identify users/stakeholders 

each requirement has a singular concept 


some requirements are compound statements 

requirements definition minimized when funding short 


program scope may reduce, but requirements definition completed 

requirements extraction has formal process 

X 

requirements extraction ad hoc 

change procedures formal 

X 

change procedures ad hoc 

users/stakeholders somehow involved in requirements definition 

X 

program team only involved in requirement definition 

management sets requirements for developers 


developers at least partially involved in setting requirements 

requirements changed at least once since baseline established prior to new version 

X 

requirements in baseline has not changed prior to new version or upgrade 

no ranking of requirements 

X 

requirements have priorities assigned 

use-case diagrams (or other models or scenario developments) 


no models used for requirements extraction 

requirements changes informal 


requirements changes formal 

plan to "freeze" requirements at some designated milestone 

X 

no provision for "freezing" requirements 

requirements must be traceable 

X 

origin of requirements not important 

requirements must be testable 

X 

system developed must be testable 

test plans to determine requirements implemented 

X 

no test plans needed for requirements verification 

requirements have priorities in implementation 


all requirements must be implemented 

some requirements have multiple statements or ideas 

X 

one idea, one statement per requirement 


Requirements Management (page 1 of 2) score | 31 
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Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


j ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) ( 

requirements first, then initial development work 


initial development work then requirements 


requirements documentation driving development 


requirements documentation developed in parallel/after development 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


change management procedures used strictly 


change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 


design decisions only after approved requirements stabilized 


requirements summarized wht we have developed 


requirements are the blueprint for development 


length of time for requirements work greater than development work 


length of time for requirements work less than development work 


requirements have design detail 


no design detail in requirements 


requirements creep to be avoided 


requirements creep o.k., but need to be controlled 


freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 


informal change procedure 


change management plan 


no change management plan 


requirements ambiguity always present to some extent 


requirements ambuiguity unacceptable at any level 


testing considered up frornt during requirements determination 


testing considered down the line during development 


requirements development team members different from implementation 


those working on requirements, work on implementation 


start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 


j ANSWER THIS BLOCK OF QUESTIONS ONL Y IF A PROTOTYPING, THROW A WA Y, SYNCHRONIZE & STABILIZE, OR OTHER STRA TEGY USED 1 

develop prototype, then determine requirements 


determine requirements prior to any development work 

X 

requirements testing done after each iteration 


no testing 

X 

individual changes as necessary 

X 

only block changes made 


development team decides on changes after iteration 

X 

users involved with changes 


changes based on feedback only from user for correction of problems 


changes to upgrade system and correct problems 

X 

funding controls changes and change procedures 

X 

changes control funding 


requirements documentation finalized prior to development 

X 

requirements fluid throughout development (only freeze at end) 


requirements test plans completed prior to development 

X 

requirements test plans completed after development 


requirements first, then initial development work 

X 

initial development work then requirements 

X 

use development effort to learn more about requirements 

X 

define all requirements prior to coding anything 


requirements ambiguity always present to some extent 

X 

requirements ambiguity unacceptable at any level 


requirements have design detail 

X 

no design detail in requirements 


user feedback considered during development 

X 

after development starts, user feedback serves as input to new work 


get something to users as soon as possible for evaluation 

X 

make sure it is complete before releasing 


management dictates requirements 

X 

development team visually represent requirements through rapid prototyping 


new requirements allowed after initial requirements defined 

X 

new requirements not allowed 



Requirements Management (pg 2 of 2) score [13] +pg 1 score [31] = TOTAL SCORE [44] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 

formal derivation of product metric for estimation of size 


ad hoc size estimation 

ad hoc process evaluation 


formal derivation of at lest one process metric 

develop work breakdown structure (WBS) 

X 

assign work as needs arise 

estimates are developed to fulfill a data call only 


use estimates to plan program 

use estimates to sell program only 


estimates are useful to the project tema for planning purposes 

resource evaluations made for program 

X 

no resource evaluation for planning 

use both bottom up & top down for estimate, use one stakeholders like 


use both bottom up & top down and evaluate significant differences 

estimates made and not updated 


estimates updated throughout program 

resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 

estimations made to fit budget 


budget made from estimations 

estimations compromised to get program 


rather risk loss of program than compromise confident estimations 

cycle time estimations 

X 

no cycle time estimations 

event count estimations 

X 

no event count estimations 

lines of code (LOC) estimation 

X 

no LOC estimation 

function pont (FP) estimation 


no FP estimation 

estimates by algorithmic methods 


estimates by analogy 

expert judgement for estimates 

X 

ad hoc estimates 

estimates by algorithmic methods 

X 

ad hoc estimates 

expert judgement for estimates 


estimates by analogy 

ad hoc estimates 


estimates by analogy 

bottom up estimates 

X 

expert judgement 

top down estimates 

X 

expert judgement 

ad hoc estimates 


any other estimate process 

fuzzy logic estimating method 


no formal estimation methodology 

WBS development from estimates 


WBS development in parallel or prior to estimation completion 

critical path of program determined 

X 

tasks developed but no path is identified 

estimators are program team members 

X 

estimators are outside program team 

management only on estimations 


all team members involved in estimation process 

estimates updated at reviews 

X 

no updates of estimates 

estimates updated at reviews 


estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 


estimate procedures change 

stakeholders are part of estimation process 


stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

X 

important to have WBS as guide, not rigid implementation 

Estimation/Planning Management (page 1 of 2) score 

29 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

X 

estimates for program initiation only 


system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as w hole 


estimates for on-gong resources needed to maintain s/w 

X 

estimates for maintenance not done 


informal re-estimates during development 


formal re-estimates at pre-defined milestones 

X 

formal re-estimates w hen amendment changing the system is introduced 

X 

informal re-estimates w hen amendment changing the system 


person in-charge of estimation walks in a managers office to get an opinion 

X 

meeting(s) organized for purpose of performing cost estimations 


factor analysis prior to commencement of program 


none done 

X 

change control procedures set in place 

X 

no set procedures 


elapsed time and actual work time estimates 


one or the other or neither 

X 

no schedule created 


scheudle created 

X 

schedule not updated 


schedule updated 

X 

schedule followed 

X 

schedule not followed 


tasks identification arises as program progresses 

X 

detailed level tasks identified prior to program initiation 


scope of program understood by all 

X 

scope not explicitly defined 


quality factors and criteria identified 


no explicit quality factors defined 

X 

no project tracking tools used 


project tracking tools used 

X 

CSCIs identified and tasked 


CSCIs not explicitly identified 

X 

expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 


no cost schedule developed 


cost schedule developed 

X 

no resource schedule developed 


resource schedule developed 

X 

team members, management know at any time if in budget & schedule 


exact budget & schedule status somew hat unclear to at least some 

X 

individual program phases are estimated 

X 

only top level program estimated 


stakeholders/users emphasis understood-quickto field or all complete 

X 

program management sets delivery tradeoffs w ithout outside input 


testing planned w ith initial program planning 

X 

testing not in initial planning 


documentation not considered ininitial planning 

X 

documentation part of initial planning 


hardw are considered in estimations 

X 

software only considered 


no formal schedule/cost tracking 


formal procedures established for tracking cost and schedule 

X 

earned value set up 

X 

earned value not used 


estimations omit documentation planning 

X 

documentation in estimates 


training omitted in estimates 

X 

training part of estimates 


earned value set up, but not tracked 

X 

earned value tracked 


detailed planning done w ith incomplete set of requirements 

X 

detailed planning done w ith detailed set of requirements 


complete infrastructure support mechanism understood for estimations 

X 

no consideration of infrastructure done for estimations 


team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 


work breakdow n structure (WBS) set up 

X 

no WBS completed 



Estimation/Planning Management (pg 2 of 2) score [23] +pg 1 score [29] = TOTAL SCORE I 52 I Enter on QMM scoresheet blk b. 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 1 of 2): 
Human Resources 


program team members have clearly deined, segmented roles 

X 

work responsibilities are shared 


formal team building procedures are used 

X 

no formal team building emphasized 


program manager flexible regarding work hours 

X 

program manager maintains strict standards for work hours 


big picture conveyed to all team members by program management 

X 

program management focuses on the partitioned tasks with team 


people issues dealt with primarily through indirect methods (email, memo, etc) 


people issues dealt with primarily through direct methods (face-to-face) 

X 

training is required and planned on a regular basis 

X 

training is ad hoc 


each team member is educated on and understands overall program and their roles 


team members only know their respective areas 

X 

consideration for team members' career goals are reflected in assignments 

X 

team members must adapt to tasks that are assigned 


team members assignments and responsibilities are mostly dictated by PM 


assignments and responsibilities are discussed and agreed upon with PM 

X 

management leads in problem solving 


management facilitates and lets team lead in problem solving 

X 

management welcomes problems as challenges and opportunities 

X 

management views problems as obstacles and grounds for punishment 


team members participate in performance evaluations of peers 

X 

Personnel evaluations are strictly PM responsibility 


management reinforcement feedback sparse and inconsistent, if any 


management provides timely reinforcement feedback for positive behaviors 

X 

management provides basic needs of office facilities fairly well 

X 

office facilities are a drawback to working in the program 


working conditions are fairly comfortable, time off policy fairly good 

X 

working conditions and time off policy is inconsistent and difficult at times 



Communication: 



communications primarily written (email) 


detailed instructions: oral presentation, follow-up email 


formal communication protocol 


external vertical communications restricted 


coders notebook weekly accomplishment reports required 


user-coder relationship established, encouraged, and mediated 


meetings structured to minimize waster time 


meetings have agenda, objectives, and conclude with action items 


program management and coder communication face to face 


program team updated regularly regarding organizational & program status 


open communications is encouraged 


program manager accessible for discussions 


program management (PM) is viewed as separate from team 


management regularly holds team meetings 


meetings are structured with definite goals and objectives 


program management is generally easy to reach and talk to 


team-program manager relationship adult-aduit 


schedules are spontaneous and poorly communicated 


work is seen as complex processes involving team working together 


action items often is poorly disseminated and usually not followed through 


team members require frequent clarifications by PM for assigned tasks 


communications primarily verbal (face-to-face) 


email only 


informal communications 


external vertical communication allowed 


not required 


user-coder interaction minimized 


meetings unstructured and open ended 


Imeeting agenda fluid and open ended 


program management and coder communication primarily email 


meetings infrequently scheduled 


communication hrough chain of command only is encouraged 


program manager difficult to get an appointment to see 


| PM mixes with team frequently 


meetings are sporadic 


meetings are informal 


PM is usually hard to get a hold of and difficult to talk to 


Iteam-program management relationship parent-child 


schedules must be fixed and rigidly followed and formally reported 


|work broken into pieces with minimal team member interaction 


action items communicated and followed through thoroughly 


team members rarly require clarifications by PM for assigned tasks 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

X 

short tern program and immediate work focus 


lead through personal attention to others 


action-oriented leadership approach 

X 

run as much of the organization as possible 


let team make decisions as much as possible 

X 

direct and domineering style 


encourage independence in others 

X 

traditional leaders respect hierarchy 


do what needs to be done 

X 

win cooperation rather than demand it 

X 

tough-minded with others 


act strongly and forcefully in the field of ideas 

X 

prefer to lead other independent types while seeking autonomy for self 


consults with team members to find solutions to problems 

X 

consults team members to get validation of PM's predetermined solutions 


keep people well informed 

X 

only as much knowledge as necessary for their work 


make things happen by focusing on the immediate problems 

X 

long range focus and de-emphasize current problems 

X 

manage others loosely and prefer minimal supervision 


follow traditional procedures and rules conscientiously 


leadership, management decisions exclusively by program management 

X 

program management makes decisions but gets inputs from team 

X 

team-program manager relationship adult-adult 


team-program management relationship parent-child 


program management makes decisions but gets inputs from team 

X 

all program team members responsible for program decisions 


when a problem arises: management takes over to solve it 

X 

management lets the team solve the problems 


leadership is do as 1 say, not do as 1 do 

X 

leadership by example 


program expectation not influenced by PM 

X 

program expectation managed by PM 


PM gives freedom to team, but has no mentoring for members (abdication) 

X 

PM empowers teams by mentoring members to be leaders 


promgram management waits and sees what happens then plans 

X 

management plans far in advance 


program management is constantly reacting to emergencies 


management is one step ahead of problems 

X 

facilitative approach to solving problems 


take charge readily and often 

X 

program management is complex, takes much time to understand 

X 

management is simple, easy to figure out 

X 

program management prefers to plunge right in 


takes time to separate things to be done and order of doing them 

X 

program management reacts spur of the moment 


methodically follows plans 

X 


Technical Competency of the Program Manager: 


PM has technical experience particular to the particular s/w program 


PM relies on team members solely 

X 

PM participates in technical reviews 

X 

PM only in non-technical reviews 


PM participates in making technical decisions when problems arise 


PM delegates technical questions 

X 

PM does not get involved discussing technical options 


PM contributes to technical options being discussed 

X 

PM does not review technical options and decisions 


PM reviews technical options and decisions 

X 

PM actively attempts to keep up-to-date with current technology and standards 

X 

PM is removed from cutting edge technology issues 


PM receives technical periodicals and occasionally references applicable articles 

X 

PM doesn't read periodicals nor reference current articles to team 


PM doesn't have technical background (or education) 


PM has technical background (or education) 

X 

team members avoid PM when they need technical advice 


team members generally consider talking to PM regarding technical issues 

X 


HR [13] + Comm. [18] + Leadership [16] + Tech. Competency [7] = People Mgmt. score [54] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 


RM is informal, if at all 

X 

a risk management plan exists 

X 

no risk management plan is developed 


RM is more of a data call than a useful document 

X 

RM drives decisions on the program 


RM is done prior to the program beginning 


RM is done prior and during program execution 

X 

RM is only done during the program execution 


RM is done prior and during program execution 

X 

risks are generalized through the whole program 


risks are categorized 

X 

risk management is done internally, only 


an outside organization also contributes to the RM process 

X 

risk is a management function 


risk is a program team function 

X 

risks are precisely articulated 


risks are generalized, if at all 

X 

each risk has a consequence 


consequences are generalized, if at all 

X 

a mitigation strategy is completed for each risk 


mitigation strategy is generalized, if at all 

X 

contingency plans are developed for a RM plan 


contingency plans are ad hoc as problems arise in the program 

X 

risks are anticipated 


if problems arise, management will deal with it 

X 

the program doesn't have any risk 


programs that do not have risk, have problems 

X 

risk management is automated 


risk management may use tools, but depend on human input 

X 

risks are assigned probabilities 

X 

probabilities are not relevant for RM 


all risks are potential problems, relative priorities for risks are not useful 


risks are weighed relative to other program risks and thus prioritized 

X 

risk management information is only shared internally 


risk management information is shared with all stakeholders 

X 

risk analysis uses ordinal rankings 

X 

risk analysis uses actual measurements with a mathematical model 


regret analysis used 


no regret analysis done 

X 

attach probabilities to future events 


no probabilities associated with future events 

X 

assessing risks with mechanical meethods 


risks should be compared to other risks and sorted 

X 

risk status tracked 

X 

not tracked 


technical risks examined 

X 

no technical risks examined 


process risks examined 

X 

no process risks examined 


product risks examined 

X 

no product risks examined 


stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 


checklists used to identify risks 

X 

no checklists used 


risks are tracked 

X 

no tracking or monitoring of risks 


each risk has an impact 


no impact analysis of risk 

X 

each risk has a mitigation plan 

X 

no individual risk mitigation 


risks monitored by priority 

X 

no special attention to track higher priority risks 


risk assessment is formalized 


no formal risk assessment 

X 

risk control is formalized 


no formal risk control 

X 

integration risks not considered 


integration risks examined 

X 


Risk Management (page 1 of 2) score 


22 


59 














































































































Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 


unforeseen risks have occurred in program 


any risk that came up had been identified previously 

X 

personnel risks examined 

X 

no personnel risks examined 


estimation risks examined 

X 

no estimation risks examined 


planning risks examined 

X 

no planning risks examined 


requirements risks examined 

X 

no requirements risks examined 


resource risks examined 

X 

no resource risks examined 


risk management plan updated regularly 

X 

no regular risk management plan updates 


risks charted 


risks not charted 

X 

performance risks examined 

X 

performance risks not examined 


program management self risks examined 


no program management risks examined 

X 

risk from program constraints examined 

X 

no program constraint risks examined 


each category of risks are prioritized 

X 

no prioritization 


each category of risks are evaluated for impact 

X 

no impact analysis performed 


each category of risks have control strategy 


no control strategy 

X 

documentation risks examined 

X 

no documentation risks examined 


regret matrix tracked 


no regret matrix or not tracked 

X 

communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 


taxonomy-based questionnaire used to identify risks 


taxonomy-based questionnaire not used 

X 

associated hardware risks examined 

X 

no consideration for hardware risks 


integration risks examined 

X 

integration risks not examined 


communication risks examined 

X 

communication risks not examined 


leadership risks examined 

X 

leadership risks not considered 


risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 


risk documentation forms used 


no risk documentation forms used 

X 

dependency risks examined 

X 

no dependency risks examined 


alternatives like risk avoidance considered for high risk items 

X 

no consideration of risk avoidance 


documented risk statements use a condition-consequence type format 


condition-consequence of risk statements not clearly defined 

X 

no assignment of ownership of risk mitigation action 


each risk mitigation action is assigned to an individual for resolution 

X 

calculation of risk exposure made (probability X loss, for each risk) 


no risk exposure calculations 

X 

oral communication of risks only 


risks written in a way that communicates nature and status of factors 

X 

triggers used to quantify risk conditions present 


risk conditions present are all subjective 

X 

risk "czar" in program for monitoring risks 


no special positions/responsibilities for risk monitoring 

X 

post-program review completed (scheduled) for unanticipated problems ID 

X 

no post-program reviews completed or scheduled 

X 

no schedule risks examined 


risks to schedule investigated 

X 


Risk Management (pg 2 of 2) score [23] +pg 1 score [22] = TOTAL SCORE [55] Enter on QMM scoresheet blk d. 
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C. PROGRAM B - PROGRAM MANAGER 
1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

62 

e 

48 

110 


0.92 

= 

101.2 

Est./Planning 

Management 

D 

66 

D 

53 

119 


0.67 

= 

79.73 

People 

Management 

c 

61 

g 

43 

104 


1.86 

= 

193.44 

Risk Management 

D 

62 

D 

54 

116 


0.55 

= 

63.8 


QMM SCORE 438.17 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 

QMM percentage score: 86.37% 


Objective/Subjective view of the overall success of program A on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Program Manager 

Success Score: 8.5 
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2 , 


Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e Yes No N/A 


1 

PM chose to have a formal requirements list 

X 



2 

Requirements recorded in someway 

IT 



3 

Written requirements were part of some formal document 

IT 



4 

Written requirements were informal 


X 


5 

At least some requirements were oral only 


“x - 


6 

All stakeholders were identified 

IT 



7 

All stakeholders participated in the requirements extraction 


~ir 


8 

Some stakeholders participated in the requirements extraction 

IT 



9 

Management extracted requirements, no stakeholder involvement 


~ir 


10 

Management passed requirements to development team 

"IT 



11 

Stakeholders not involvved in Management extraction, but approved 


~ir 


12 

Management gets inputs from stakeholders, then develops requirements 

~ 



13 

Developers w ork informally w ith users to arrive at requirements 


~ 


14 

Same as 13, but management oversees and formalizes 

IT 



If a waterfall or sequential development strategy: [ 

15 

All requirements complete before design 




16 

Some requirements left incomplete prior to design 




17 

Requirements informal prior to design effort 




18 

Requirements serve as input 




19 

Length of time for requirements w ork greater than development w ork 




20 

Requirements developed in parallel to design 




OR If a prototype, throwaway, or other development strategy: | 

15 

Learn about requirements through development efforts 

X 



16 

No coding until all requirements are defined 


X 


17 

Requirements formal prior to design effort 


“x - 


18 

Requirements serve as output 

X 



19 

Requirements definition work in parallel to development efforts 

X 



20 

Requirements developed in parallel to design 

X 



21 

Are requirements frozen at some phase 


~x~ 


22 

Change management exists 

X 



23 

Change management is formal 

X 



24 

Project strategy is consistent throughout development 

X 



25 

Requirements are updated 

X 



26 

Configuration Management (CM) exists 

X 



27 

CM is formal 

X 



28 

Requirements are testable 

X 



29 

Requirements testing considered/implemented during extraction 

X 



30 

Requirements testing plan exists 

X 



31 

Requirements testing is formal 

X 



32 

All requirements have priorities 

X 



33 

All requirements must be implemented 


— 


34 

Requirements are tested 

X 



35 

All requirements are equally important 


— 


36 

At least some requirements have priorities 

X 



37 

All requirements are traceable 

X 



38 

Traceability not important 


— 


39 

Each requirement has an author 


— 


40 

Who authored requirement is not important 


“x - 


41 

Initial set of requirements to be implemented, no requirements creep 

X 



42 

Structured and tracked changes to requirements only 

X 



43 

Change is inevitable, changes allowed at all times 


— 


44 

Change is inevitable, but changes limited 

X 



45 

Requirements control funding 

X 



46 

Requirements history kept 

X 



47 

Baseline established for requirements at some point prior to develop 

X 



TOTAL SCORING 

m 

D 

0 
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3 


Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f Yes No N/A 


1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 


X 


4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

X 



5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 

X 



7 

Time measure process metric used (cycle time) 

X 



8 

Process metrics tracked and updated throughout program execution 

X 



9 

Program cost estimations made from product or process metrics 

X 



10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 

X 



12 

Program's primary purpose, including major functions and deliverables known 

X 



13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 




15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 




18 

Quality assurance plan or similar to aid in detecting defects early in program 




19 

COCOMO estimates performed 




20 

CSCI clearly defined and tasked 

X 



21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 

X 



29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 

X 



33 

Automated program tracking used 

X 



34 

PM usually thorough in tracking and reporting schedules and financials 

X 



35 

WBS developed only as data call 


X 


36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 

X 



38 

Estimations are done using both top down and bottoms up approaches 

X 



39 

All program team members involved in planning process 

X 



40 

Plardware also considered in estimaation process 

X 



41 

Program history compiled 

X 



42 

System upgrades (SCR) software changes requests estimated individually 

X 



43 

Management duties apart of each team member's responsibilities 


X 


44 

PM dictates schedules to program team 


X 


45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 

X 



47 

Test planning done at the start of the program 

X 



48 

Estimations are completed by those performing the tasks 

X 



49 

Sensitivity analysis performed for program choices 

X 



50 

Software deployment planning completed 

X 



TOTAL SCORING 

m 

T" 
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4, 


People Management Questionnaire Responses 


No. People Management Questionnaire - Total: Block g Yes No N/A 


1 

PM is accessible in person by each team member 

X 



2 

PM is accessible via email (memo, letter) by each team member 

X 



3 

PM is accessible via phone by each team member 

X 



4 

PM not only considers a person's suitability, not also desire to be on a team 

X 



5 

PM consults with each team member regarding their career goals 


X 


6 

PM regularly holds meetings to inform team of program progress 


X 


7 

PM solicits opinions from team members before making decisions 

X 



8 

PM lets teams make decisions affecting their work 

X 



9 

PM freuently makes decisions without any consultation with members 


X 


10 

PM understands the technology/language of the program 

X 



11 

PM is able to communicate with other the technical issues in the program 

X 



12 

PM prioritized problems or conflicts within the program 

X 



13 

PM assists team members in developing/advising of career path 

X 



14 

PM empowers program members to recommend hiring new team members 

X 



15 

PM empowers program members to recommend firings of other members 

X 



16 

PM specifically assigns work to each program member 


X 


17 

PM sets communication protocol 

X 



18 

PM allows unrestricted communications 

X 



19 

PM encourages or requires training for each individual 

X 



20 

PM takes control in difficult/roblem areas 

X 



21 

PM looks ahead to new programs, new upgrades of existing program 

X 



22 

PM maintains regular communications with all stakeholders 

X 



23 

PM maintains regular communications with users 

X 



24 

PM encourages program team communication with users 

X 



25 

PM encourages program team communication with stakeholders 

X 



26 

PM facilitates horizontal communication within program 

X 



27 

PM facilitates communication during integration 

X 



28 

PM holds meetings without clear objectives 

X 



29 

PM must approve all decisions within the program 

X 



30 

PM must approve all interactions with stakeholders 

X 



31 

PM must approve all interactions with users 

X 



32 

PM makes all presentations to stakeholders/users 


X 


33 

PM is considered "flexible" in terms of program members personal issues 

X 



34 

PM, at least occasionally, schedules/promotes outside work team activities 

X 



35 

PM is readily willing to listen to program prblems and complaints 

X 



36 

PM takes action to resolve program problems and complaints 

X 



37 

PM is generally respected by stakeholders, users, and organization 

X 



38 

PM sometimes fails to grasp important technical issues in program 

X 



39 

PM recruits program team members from outside organization 

X 



40 

PM participates in technical reviews 

X 



41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual's tasks are specific, each exposed to the "bigger picture" 

X 



43 

PM has clearly defined his/her expectations for each individual 

X 



44 

PM delegation of duties is usually seemless in execution 

X 



45 

PM acts as facilitator to solving personnel conflicts 

X 



46 

PM attempts to motivate individuals on the program team 

X 



47 

PM clearly spearates technical from managerial roles for individuals 

X 



48 

PM directs how he/she expects the task to be accomplished 

X 



49 

PM directs what needs to be done, but does not direct how 

X 



50 

PM attempts to spotlight individuals in the program for positive exposure 

X 



TOTAL SCORING 

EH 

“7" 
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Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Total: Block h Yes No N/A 


1 

Risk Management (RM) is specifically an activity in the program 

X 



2 

RM is formal and documented 

X 



3 

A specific RM Ian exists 

X 



4 

RM is required in the program, but not used during the program 

X 



5 

RM is done prior to the program execution 

X 



6 

RM is done by an outside entity to the development 

X 



7 

RM is done internally only 


X 


8 

RM is both internally performed and externally assessed 

X 



9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is only a management function 


X 


11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 

X 



13 

Risks are only generalized 

X 



14 

Each risk is delineated 

X 



15 

Each risk has a consequence 

X 



16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 

X 



18 

Risk Management is automated 

X 



19 

Risks are tracked 

X 



20 





21 

Regret analysis performed 

X 



22 

RM drives decisions in the program 

X 



23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 

X 



25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 

X 



28 

Risk Assessment done prior to program start 

X 



29 

Risk Assessment includes personnal risk 

X 



30 

RM uses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 

X 



34 

Risk Assessment is briefed organization structure above program manager 

X 



35 

Risk Assessment includes requirements risks 

X 



36 

Risk Assessment includes user risks (too little involvement of user) 

X 



37 

Risk Assessment includes documentation risks 

X 



38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 

X 



42 

Documentation proof exists to demonstrate following risk management plan 

X 



43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 

X 



45 

Other organizational checklists used for risk assessment 

X 



46 

Internal organizational checklists used for risk assessment 

X 



47 

Risk Assessment information contributed to internal or other database 

X 



48 

Risk Assessment includes internal organization risks 

X 



49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 

HrTnBf 

B 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

X 

informal requirement list 

w ritten requirements 

X 

oral requirements 

requirements informal, but recorded 

X 

requirements not recorded 

requirements as part of an SRS (or other formal repository) 

X 

requirements informally recorded 

requirements taken as is from customer 

X 

look to reformulate, interview in-depth, or otherwise re-validate 

only one development strategy used 


strategies not consistent, used at different times 

stakeholders as part of requirements development 

X 

stakeholders approving requirements after formulated by development team 

requirements are testable 

X 

requirements have no test plans 

informal test plan or no test plan 


formal test plan 

test team involved w ith requirements 

X 

no test team input or plans during requirements development 

only a percentage of requirements present in baseline 

X 

baseline must contain all requirements 

requirements documentation has hierarchical structure 

X 

all requirements must be implemented 

requirements have listed responsible party 

X 

requirements origin not important 

requirements documentation have versions 

X 

no requirements history 

requirements have specific attribute values 

X 

requirements all rank evenly 

funding controls requirements definition 

X 

requirements definition controls funding 

reqquirements are top dow n 

X 

requirements are bottom up 

users/stakeholders are identified and interviewed (market survey) 


no special consideration to identify users/stakeholders 

each requirement has a singular concept 

X 

some requirements are compound statements 

requirements definition minimized when funding short 

X 

program scope may reduce, but requirements definition completed 

requirements extraction has formal process 

X 

requirements extraction ad hoc 

change procedures formal 

X 

change procedures ad hoc 

users/stakeholders somehow involved in requirements definition 

X 

program team only involved in requirement definition 

management sets requirements for developers 

X 

developers at least partially involved in setting requirements 

requirements changed at least once since baseline established prior to new version 

X 

requirements in baseline has not changed prior to new version or upgrade 

no ranking of requirements 

X 

requirements have priorities assigned 

use-case diagrams (or other models or scenario developments) 

X 

no models used for requirements extraction 

requirements changes informal 

X 

requirements changes formal 

plan to "freeze" requirements at some designated milestone 


no provision for "freezing" requirements 

requirements must be traceable 

X 

origin of requirements not important 

requirements must be testable 


system developed must be testable 

test plans to determine requirements implemented 

X 

no test plans needed for requirements verification 

requirements have priorities in implementation 

X 

all requirements must be implemented 

some requirements have multiple statements or ideas 


one idea, one statement per requirement 


Requirements Management (page 1 of 2) score | 44 
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Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


] ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) ( 

requirements first, then initial development work 


initial development work then requirements 


requirements documentation driving development 


requirements documentation developed in parallel/after development 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


change management procedures used strictly 


change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 


design decisions only after approved requirements stabilized 


requirements summarized wht we have developed 


requirements are the blueprint for development 


length of time for requirements work greater than development work 


length of time for requirements work less than development work 


requirements have design detail 


no design detail in requirements 


requirements creep to be avoided 


requirements creep o.k., but need to be controlled 


freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 


informal change procedure 


change management plan 


no change management plan 


requirements ambiguity always present to some extent 


requirements ambuiguity unacceptable at any level 


testing considered up frornt during requirements determination 


testing considered down the line during development 


requirements development team members different from implementation 


those working on requirements, work on implementation 


start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 


j ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWAY, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 1 

develop prototype, then determine requirements 

X 

determine requirements prior to any development work 


requirements testing done after each iteration 

X 

no testing 


individual changes as necessary 

X 

only block changes made 


development team decides on changes after iteration 


users involved with changes 

X 

changes based on feedback only from user for correction of problems 

X 

changes to upgrade system and correct problems 


funding controls changes and change procedures 

X 

changes control funding 


requirements documentation finalized prior to development 


requirements fluid throughout development (only freeze at end) 

X 

requirements test plans completed prior to development 

X 

requirements test plans completed after development 


requirements first, then initial development work 


initial development work then requirements 

X 

use development effort to learn more about requirements 

X 

define all requirements prior to coding anything 


requirements ambiguity always present to some extent 

X 

requirements ambiguity unacceptable at any level 


requirements have design detail 

X 

no design detail in requirements 


user feedback considered during development 

X 

after development starts, user feedback serves as input to new work 


get something to users as soon as possible for evaluation 

X 

make sure it is complete before releasing 


management dictates requirements 


development team visually represent requirements through rapid prototyping 

X 

new requirements allowed after initial requirements defined 

X 

new requirements not allowed 



Requirements Management (pg 2 of 2) score [18] +pg 1 score [44] = TOTAL SCORE [62] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 

formal derivation of product metric for estimation of size 

X 

ad hoc size estimation 

ad hoc process evaluation 


formal derivation of at lest one process metric 

develop w ork breakdow n structure (WBS) 

X 

assign work as needs arise 

estimates are developed to fulfill a data call only 


use estimates to plan program 

use estimates to sell program only 


estimates are useful to the project terra for planning purposes 

resource evaluations made for program 

X 

no resource evaluation for planning 

use both bottom up & top dow n for estimate, use one stakeholders like 


use both bottom up & top dow n and evaluate significant differences 

estimates made and not updated 


estimates updated throughout program 

resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 

estimations made to fit budget 


budget made from estimations 

estimations compromised to get program 


rather risk loss of program than compromise confident estimations 

cycle time estimations 

X 

no cycle time estimations 

event count estimations 

X 

no event count estimations 

lines of code (LOC) estimation 

X 

no LOC estimation 

function pont (FP) estimation 

X 

no FP estimation 

estimates by algorithmic methods 

X 

estimates by analogy 

expert judgement for estimates 

X 

ad hoc estimates 

estimates by algorithmic methods 

X 

ad hoc estimates 

expert judgement for estimates 


estimates by analogy 

ad hoc estimates 


estimates by analogy 

bottom up estimates 

X 

expert judgement 

top dow n estimates 

X 

expert judgement 

ad hoc estimates 


any other estimate process 

fuzzy logic estimating method 

X 

no formal estimation methodology 

WBS development from estimates 

X 

WBS development in parallel or prior to estimation completion 

critical path of program determined 

X 

tasks developed but no path is identified 

estimators are program team members 

X 

estimators are outside program team 

management only on estimations 


all team members involved in estimation process 

estimates updated at review s 

X 

no updates of estimates 

estimates updated at review s 


estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 

X 

estimate procedures change 

stakeholders are part of estimation process 

X 

stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

X 

important to have WBS as guide, not rigid implementation 

Estimation/Planning Management (page 1 of 2) score 1 33 



Estimation/Planning Management (page 1 of 2) score 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

X 

estimates for program initiation only 

system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as w hole 

estimates for on-gong resources needed to maintain s/w 

X 

estimates for maintenance not done 

informal re-estimates during development 


formal re-estimates at pre-defined milestones 

formal re-estimates w hen amendment changing the system is introduced 

X 

informal re-estimates w hen amendment changing the system 

person in-charge of estimation walks in a managers office to get an opinion 


meeting(s) organized for purpose of performing cost estimations 

factor analysis prior to commencement of program 

X 

none done 

change control procedures set in place 

X 

no set procedures 

elapsed time and actual work time estimates 

X 

one or the other or neither 

no schedule created 


scheudle created 

schedule not updated 


schedule updated 

schedule follow ed 

X 

schedule not followed 

tasks identification arises as program progresses 


detailed level tasks identified prior to program initiation 

scope of program understood by all 

X 

scope not explicitly defined 

quality factors and criteria identified 

X 

no explicit quality factors defined 

no project tracking tools used 


project tracking tools used 

CSCIs identified and tasked 

X 

CSCIs not explicitly identified 

expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 

no cost schedule developed 


cost schedule developed 

no resource schedule developed 


resource schedule developed 

team members, management know at any time if in budget & schedule 

X 

exact budget & schedule status somew hat unclear to at least some 

individual program phases are estimated 

X 

only top level program estimated 

stakeholders/users emphasis understood-quickto field or all complete 

X 

program management sets delivery tradeoffs w ithout outside input 

testing planned with initial program planning 

X 

testing not in initial planning 

documentation not considered ininitial planning 


documentation part of initial planning 

hardw are considered in estimations 

X 

software only considered 

no formal schedule/cost tracking 


formal procedures established for tracking cost and schedule 

earned value set up 

X 

earned value not used 

estimations omit documentation planning 


documentation in estimates 

training omitted in estimates 

X 

training part of estimates 

earned value set up, but not tracked 


earned value tracked 

detailed planning done w ith incomplete set of requirements 

X 

detailed planning done w ith detailed set of requirements 

complete infrastructure support mechanism understood for estimations 

X 

no consideration of infrastructure done for estimations 

team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 

work breakdow n structure (WBS) set up 

X 

no WBS completed 


Estimation/Planning Management (pg 2 of 2) score [33] +pg 1 score [33] = TOTAL SCORE [66] Enter on QMM scoresheet blk b. 
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Pair choice section THREE: 
Human Resources 


(People Management) choose most applicable term of the two for each row (page 1 of 2): 


program team members have clearly deined, segmented roles 


work responsibilities are shared 


formal team building procedures are used 


no formal team building emphasized 


program manager flexible regarding work hours 


program manager maintains strict standards for work hours 


big picture conveyed to all team members by program management 


program management focuses on the partitioned tasks with team 


people issues dealt with primarily through indirect methods (email, memo, etc) 


people issues dealt with primarily through direct methods (face-to-face) 


training is required and planned on a regular basis 


training is ad hoc 


each team member is educated on and understands overall program and their roles 


team members only know their respective areas 


consideration for team members' career goals are reflected in assignments 


team members must adapt to tasks that are assigned 


team members assignments and responsibilities are mostly dictated by PM 


management leads in problem solving _ 

management welcomes problems as challenges and opportunities 


assignments and responsibilities are discussed and agreed upon with PM 


management facilitates and lets team lead in problem solving 


team members participate in performance evaluations of peers 


management views problems as obstacles and grounds for punishment 


Personnel evaluations are strictly PM responsibility 


management reinforcement feedback sparse and inconsistent, if any 


management provides timely reinforcement feedback for positive behaviors 


management provides basic needs of office facilities fairly well 


office facilities are a drawback to working in the program 


working conditions are fairly comfortable, time off policy fairly good 


working conditions and time off policy is inconsistent and difficult at times 


Communication: 


communications primarily written (email) 

X 

communications primarily verbal (face-to-face) 


detailed instructions: oral presentation, follow-up email 

X 

email only 


formal communication protocol 

X 

informal communications 


external vertical communications restricted 


external vertical communication allowed 


coders notebook weekly accomplishment reports required 


not required 


user-coder relationship established, encouraged, and mediated 

X 

user-coder interaction minimized 


meetings structured to minimize waster time 

X 

meetings unstructured and open ended 


meetings have agenda, objectives, and conclude with action items 

X 

meeting agenda fluid and open ended 


program management and coder communication face to face 

X 

program management and coder communication primarily email 


program team updated regularly regarding organizational & program status 

X 

meetings infrequently scheduled 


open communications is encouraged 

X 

communication hrough chain of command only is encouraged 


program manager accessible for discussions 

X 

program manager difficult to get an appointment to see 


program management (PM) is viewed as separate from team 


PM mixes with team frequently 


management regularly holds team meetings 


meetings are sporadic 


meetings are structured with definite goals and objectives 

X 

meetings are informal 


program management is generally easy to reach and talk to 

X 

PM is usually hard to get a hold of and difficult to talk to 


team-program manager relationship adult-aduit 

X 

team-program management relationship parent-child 


schedules are spontaneous and poorly communicated 

X 

schedules must be fixed and rigidly followed and formally reported 


work is seen as complex processes involving team working together 

X 

work broken into pieces with minimal team member interaction 


action items often is poorly disseminated and usually not followed through 


action items communicated and followed through thoroughly 


team members require frequent clarifications by PM for assigned tasks 


team members rarly require clarifications by PM for assigned tasks 



XX XX XX XX XX 


















































































































Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

X 

short tern program and immediate w ork focus 


lead through personal attention to others 


action-oriented leadership approach 

X 

run as much of the organization as possible 


let team make decisions as much as possible 

X 

direct and domineering style 


encourage independence in others 

X 

traditional leaders respect hierarchy 


do w hat needs to be done 

X 

w in cooperation rather than demand it 

X 

tough-minded with others 


act strongly and forcefully in the field of ideas 


prefer to lead other independent types while seeking auto no my for self 

X 

consults w ith team members to find solutions to problems 

X 

consults team members to get validation of PMs predetermined solutions 


keep people w ell informed 

X 

only as much know ledge as necessary for their w ork 


make things happen by focusing on the immediate problems 

X 

long range focus and de-emphasize current problems 


manage others loosely and prefer minimal supervision 

X 

follow traditional procedures and rules conscientiously 


leadership, management decisions exclusively by program management 


program management makes decisions but gets inputs from team 

X 

team-program manager relationship adult-adult 

X 

team-program management relationship parent-child 


program management makes decisions but gets inputs from team 

X 

all program team members responsible for program decisions 


w hen a problem arises: management takes over to solve it 


management lets the team solve the problems 

X 

leadership is do as 1 say, not do as 1 do 


leadership by example 

X 

program expectation not influenced by PM 


program expectation managed by FM 

X 

PM gives freedom to team, but has no mentoring for members (abdication) 

X 

PM empowers teams by mentoring members to be leaders 


promgram management waits and sees what happens then plans 


management plans far in advance 

X 

program management is constantly reacting to emergencies 


management is one step ahead of problems 

X 

facilitative approach to solving problems 


take charge readily and often 

X 

program management is complex, takes much time to understand 

X 

management is simple, easy to figure out 


program management prefers to plunge right in 


takes time to separate things to be done and order of doing them 

X 

program management reacts spur of the moment 


methodically follows plans 

X 


Technical Competency of the Program Manager: 


PM has technical experience particular to the particular s/w program 

X 

PM relies on team members solely 


PM participates in technical review s 

X 

PM only in non-technical reviews 


PM participates in making technical decisions w hen problems arise 

X 

PM delegates technical questions 


PM does not get involved discussing technical options 


PM contributes to technical options being discussed 

X 

PM does not review technical options and decisions 


PM review s technical options and decisions 

X 

PM actively attempts to keep up-to-date with current technology and standards 

X 

PM is removed from cutting edge technology issues 


PM receives technical periodicals and occasionally references applicable articles 

X 

PM doesn't read periodicals nor reference current articles to team 


PM doesn't have technical background (or education) 


PM has technical background (or education) 

X 

team members avoid PM w hen they need technical advice 


team members generally consider talking to PM regarding technical issues 

X 


HR [13] + Comm. [18] + Leadership [21] + Tech. Competency [9] = People Mgmt. score [61] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

X 

RM is informal, if at all 

a risk management plan exists 

X 

no risk management plan is developed 

RM is more of a data call than a useful document 


RM drives decisions on the program 

RM is done prior to the program beginning 


RM is done prior and during program execution 

RM is only done during the program execution 


RM is done prior and during program execution 

risks are generalized through the w hole program 


risks are categorized 

risk management is done internally, only 


an outside organization also contributes to the RM process 

risk is a management function 


risk is a program team function 

risks are precisely articulated 

X 

risks are generalized, if at all 

each risk has a consequence 

X 

consequences are generalized, if at all 

a mitigation strategy is completed for each risk 

X 

mitigation strategy is generalized, if at all 

contingency plans are developed for a RM plan 

X 

contingency plans are ad hoc as problems arise in the program 

risks are anticipated 


if problems arise, management w ill deal w ith it 

the program doesn't have any risk 


programs that do not have risk, have problems 

risk management is automated 

X 

risk management may use tools, but depend on human input 

risks are assigned probabilities 

X 

probabilities are not relevant for RM 

all risks are potential problems, relative priorities for risks are not useful 


risks are weighed relative to other program risks and thus prioritized 

risk management information is only shared internally 


risk management information is shared with all stakeholders 

risk analysis uses ordinal rankings 


risk analysis uses actual measurements with a mathematical model 

regret analysis used 

X 

no regret analysis done 

attach probabilities to future events 

X 

no probabilities associated with future events 

assessing risks with mechanical meethods 


risks should be compared to other risks and sorted 

risk status tracked 

X 

not tracked 

technical risks examined 

X 

no technical risks examined 

process risks examined 

X 

no process risks examined 

product risks examined 

X 

no product risks examined 

stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 

checklists used to identify risks 

X 

no checklists used 

risks are tracked 

X 

no tracking or monitoring of risks 

each risk has an impact 

X 

no impact analysis of risk 

each risk has a mitigation plan 

X 

no individual risk mitigation 

risks monitored by priority 

X 

no special attention to track higher priority risks 

risk assessment is formalized 


no formal risk assessment 

risk control is formalized 


no formal risk control 

integration risks not considered 


integration risks examined 


Risk Management (page 1 of 2) score | 33 | 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 


unforeseen risks have occurred in program 

X 

any risk that came up had been identified previously 


personnel risks examined 

X 

no personnel risks examined 


estimation risks examined 

X 

no estimation risks examined 


planning risks examined 

X 

no planning risks examined 


requirements risks examined 

X 

no requirements risks examined 


resource risks examined 

X 

no resource risks examined 


risk management plan updated regularly 

X 

no regular risk management plan updates 


risks charted 

X 

risks not charted 


performance risks examined 

X 

performance risks not examined 


program management self risks examined 

X 

no program management risks examined 


risk from program constraints examined 

X 

no program constraint risks examined 


each category of risks are prioritized 

X 

no prioritization 


each category of risks are evaluated for impact 

X 

no impact analysis performed 


each category of risks have control strategy 

X 

no control strategy 


documentation risks examined 

X 

no documentation risks examined 


regret matrix tracked 

X 

no regret matrix or not tracked 


communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 


taxonomy-based questionnaire used to identify risks 

X 

taxonomy-based questionnaire not used 


associated hardware risks examined 

X 

no consideration for hardware risks 


integration risks examined 

X 

integration risks not examined 


communication risks examined 

X 

communication risks not examined 


leadership risks examined 


leadership risks not considered 

X 

risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 


risk documentation forms used 

X 

no risk documentation forms used 


dependency risks examined 

X 

no dependency risks examined 


alternatives like risk avoidance considered for high risk items 

X 

no consideration of risk avoidance 


documented risk statements use a condition-consequence type format 

X 

condition-consequence of risk statements not clearly defined 


no assignment of ownership of risk mitigation action 

X 

each risk mitigation action is assigned to an individual for resolution 


calculation of risk exposure made (probability X loss, for each risk) 

X 

no risk exposure calculations 


oral communication of risks only 


risks written in a way that communicates nature and status of factors 

X 

triggers used to quantify risk conditions present 


risk conditions present are all subjective 

X 

risk "czar" in program for monitoring risks 


no special positions/responsibilities for risk monitoring 

X 

post-program review completed (scheduled) for unanticipated problems ID 

X 

no post-program reviews completed or scheduled 


no schedule risks examined 


risks to schedule investigated 

X 


Risk Management (pg 2 of 2) score [29] +pg 1 score [33] = TOTAL SCORE [62] Enter on QMM scoresheet blk d. 
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D. PROGRAM B - ASSOCIATE 


1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

60 

e 

47 

107 


0.92 

= 

98.44 

Est./Planning 

Management 

D 

64 

D 

52 

116 


0.67 

= 

77.72 

People 

Management 

c 

60 

g 

42 

102 


1.86 

= 

189.72 

Risk Management 

D 

61 

D 

53 

114 


0.55 

= 

62.7 


QMM SCORE 428.58 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 

QMM percentage score: 84.91% 


Objective/Subjective view of the overall success of program B on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Associate 
Success Score: 8.5 
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2 , 


Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e Yes No N/A 


1 

PM chose to have a formal requirements list 

X 



2 

Requirements recorded in some w ay 

X 



3 

Written requirements were part of some formal document 

X 



4 

Written requirements were informal 

X 



5 

At least some requirements w ere oral only 


X 


6 

All stakeholders were identified 

X 



7 

All stakeholders participated in the requirements extraction 

X 



8 

Some stakeholders participated in the requirements extraction 

X 



9 

Management extracted requirements, no stakeholder involvement 


IT 


10 

Management passed requirements to development team 

X 



11 

Stakeholders not involvved in Management extraction, but approved 


“x - 


12 

Management gets inputs from stakeholders, then develops requirements 

X 



13 

Developers w ork informally w ith users to arrive at requirements 

X 



14 

Same as 13, but management oversees and formalizes 

X 



If a wa ter fa II or sequen tia 1 devel op men t stra tegy: j 

15 

All requirements complete before design 




16 

Some requirements left incomplete prior to design 




17 

Requirements informal prior to design effort 




18 

Requirements serve as input 




19 

Length of time for requirements work greater than development work 




20 

Requirements developed in parallel to design 




OR If a prototype, throwaway, or other development strategy: | 

15 

Learn about requirements through development efforts 

X 



16 

No coding until all requirements are defined 


x 


17 

Requirements formal prior to design effort 



X 

18 

Requirements serve as output 




19 

Requirements definition work in parallel to development efforts 




20 

Requirements developed in parallel to design 




21 

Are requirements frozen at some phase 



~x~ 

22 

Change management exists 

“>T 



23 

Change management is formal 

“X - 



24 

Project strategy is consistent throughout development 

“x - 



25 

Requirements are updated 

~>r 



26 

Configuration Management (CM) exists 

~>r 



27 

CM is formal 

~>r 



28 

Requirements are testable 

~>r 



29 

Requirements testing considered/implemented during extraction 

“x - 



30 

Requirements testing plan exists 

“x - 



31 

Requirements testing is formal 

“x - 



32 

All requirements have priorities 

“x - 



33 

All requirements must be implemented 


IT 


34 

Requirements are tested 



“x - 

35 

All requirements are equally important 



“x - 

36 

At least some requirements have priorities 



“x - 

37 

All requirements are traceable 



“x - 

38 

Traceability not important 



“x - 

39 

Each requirement has an author 



“x - 

40 

Who authored requirement is not important 



“x - 

41 

Initial set of requirements to be implemented, no requirements creep 



“x - 

42 

Structured and tracked changes to requirements only 



~ir 

43 

Change is inevitable, changes allowed at all times 



“x - 

44 

Change is inevitable, but changes limited 

~ir 



45 

Requirements control funding 

~ir 



46 

Requirements history kept 

~ 



47 

Baseline established for requirements at some point prior to develop 



— 

TOTAL SCORING 

m 
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3. Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f 

Yes 

No 

N/A 

1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 

X 



4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

X 



5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 

X 



7 

Time measure process metric used (cycle time) 

X 



8 

Process metrics tracked and updated throughout program execution 

X 



9 

Program cost estimations made from product or process metrics 

X 



10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 

X 



12 

Program's primary purpose, including major functions and deliverables known 

X 



13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 

X 



15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 

X 



18 

Quality assurance plan or similar to aid in detecting defects early in program 

X 



19 

COCOMO estimates performed 



— 

20 

CSCI clearlydefined and tasked 



X 

21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 

X 



29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 


X 


33 

Automated program tracking used 


X 


34 

PM usually thorough in tracking and reporting schedules and financials 

X 



35 

WBS developed only as data call 


X 


36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 

X 



38 

Estimations are done using both top down and bottoms up approaches 

X 



39 

All program team members involved in planning process 

X 



40 

Hardware also considered in estimaation process 



— 

41 

Program history compiled 

X 



42 

System upgrades (SCR) software changes requests estimated individually 


X 


43 

Management duties apart of each team member's responsibilities 

X 



44 

PM dictates schedules to program team 


X 


45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 

X 



47 

Test planning done at the start of the program 

X 



48 

Estimations are completed bythose performing the tasks 

X 



49 

Sensitivity analysis performed for program choices 

X 



50 

Software deployment planning completed 

X 



TOTAL SCORING 

m 

"T" 
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People Management Questionnaire Responses 


No. People Management Questionnaire - Total: Block g Yes No N/A 


1 

PM is accessible in person by each team member 

X 



2 

PM is accessible via email (memo, letter) by each team member 

X 



3 

PM is accessible via phone by each team member 

X 



4 

PM not only considers a person's suitability, not also desire to be on a team 

X 



5 

PM consults with each team member regarding their career goals 

X 



6 

PM regularly holds meetings to inform team of program progress 

X 



7 

PM solicits opinions from team members before making decisions 

X 



8 

PM lets teams make decisions affecting their work 

X 



9 

PM freuently makes decisions without any consultation with members 

X 



10 

PM understands the technology/language of the program 

X 



11 

PM is able to communicate with other the technical issues in the program 

X 



12 

PM prioritized problems or conflicts within the program 

X 



13 

PM assists team members in developing/advising of career path 

X 



14 

PM empowers program members to recommend hiring new team members 

X 



15 

PM empowers program members to recommend firings of other members 

X 



16 

PM specifically assigns work to each program member 

X 



17 

PM sets communication protocol 

X 



18 

PM allows unrestricted communications 

X 



19 

PM encourages or requires training for each individual 

X 



20 

PM takes control in difficult/roblem areas 

X 



21 

PM looks ahead to new programs, new upgrades of existing program 

X 



22 

PM maintains regular communications with all stakeholders 

X 



23 

PM maintains regular communications with users 

X 



24 

PM encourages program team communication with users 

X 



25 

PM encourages program team communication with stakeholders 

X 



26 

PM facilitates horizontal communication within program 

X 



27 

PM facilitates communication during integration 

X 



28 

PM holds meetings without clear objectives 


X 


29 

PM must approve all decisions within the program 


X 


30 

PM must approve all interactions with stakeholders 


X 


31 

PM must approve all interactions with users 


X 


32 

PM makes all presentations to stakeholders/users 


X 


33 

PM is considered "flexible" in terms of program members personal issues 

X 



34 

PM, at least occasionally, schedules/promotes outside work team activities 

X 



35 

PM is readily willing to listen to program prblems and complaints 

X 



36 

PM takes action to resolve program problems and complaints 

X 



37 

PM is generally respected by stakeholders, users, and organization 

X 



38 

PM sometimes fails to grasp important technical issues in program 

X 



39 

PM recruits program team members from outside organization 

X 



40 

PM participates in technical reviews 

X 



41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual's tasks are specific, each exposed to the "bigger picture" 

X 



43 

PM has clearly defined his/her expectations for each individual 


X 


44 

PM delegation of duties is usually seemless in execution 


X 


45 

PM acts as facilitator to solving personnel conflicts 


X 


46 

PM attempts to motivate individuals on the program team 


X 


47 

PM clearly spearates technical from managerial roles for individuals 


X 


48 

PM directs how he/she expects the task to be accomplished 


X 


49 

PM directs what needs to be done, but does not direct how 

X 



50 

PM attempts to spotlight individuals in the program for positive exposure 

X 

X 


TOTAL SCORING 

m 

H 
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Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Total: Block h Yes No N/A 


1 

Risk Management (RM) is specifically an activity in the program 

X 



2 

RM is formal and documented 

X 



3 

A specific RM Ian exists 

X 



4 

RM is required in the program, but not used during the program 


X 


5 

RM is done prior to the program execution 

X 



6 

RM is done by an outside entity to the development 

X 



7 

RM is done internally only 


X 


8 

RM is both internally performed and externally assessed 

X 



9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is only a management function 


X 


11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 

X 



13 

Risks are only generalized 


X 


14 

Each risk is delineated 

X 



15 

Each risk has a consequence 

X 



16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 

X 



18 

Risk Management is automated 

X 



19 

Risks are tracked 

X 



20 





21 

Regret analysis performed 

X 



22 

RM drives decisions in the program 

X 



23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 


X 


25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 

X 



28 

Risk Assessment done prior to program start 

X 



29 

Risk Assessment includes personnal risk 


X 


30 

RM uses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 


X 


34 

Risk Assessment is briefed organization structure above program manager 

X 



35 

Risk Assessment includes requirements risks 

X 



36 

Risk Assessment includes user risks (too little involvement of user) 

X 



37 

Risk Assessment includes documentation risks 


X 


38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 

X 



42 

Documentation proof exists to demonstrate following risk management plan 

X 



43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 


X 


45 

Other organizational checklists used for risk assessment 


X 


46 

Internal organizational checklists used for risk assessment 


X 


47 

Risk Assessment information contributed to internal or other database 


X 


48 

Risk Assessment includes internal organization risks 


X 


49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 


7 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

1 x | 

informal requirement list 

written requirements 

mm 

oral requirements 

requirements informal, but recorded 

mu 

requirements not recorded 

requirements as part of an SRS (or other formal repository) 

X 

requirements informally recorded 

requirements taken as is from customer 


look to reformulate, interview in-depth, or otherwise re-validate 

only one development strategy used 

mu 

strategies not consistent, used at different times 

stakeholders as part of requirements development 

mm 

stakeholders approving requirements after formulated by development team 

requirements are testable 

mm 

requirements have no test plans 

informal test plan or no test plan 


formal test plan 

test team involved with requirements 

mm 

no test team input or plans during requirements development 

only a percentage of requirements present in baseline 


baseline must contain all requirements 

requirements documentation has hierarchical structure 

mm 

all requirements must be implemented 

requirements have listed responsible party 

mu 

requirements origin not important 

requirements documentation have versions 

1 X 1 

no requirements history 

requirements have specific attribute values 

mm 

requirements all rank evenly 

funding controls requirements definition 


requirements definition controls funding 

reqquirements are top down 

mm 

requirements are bottom up 

users/stakeholders are identified and interviewed (market survey) 

1 X 1 

no special consideration to identify users/stakeholders 

each requirement has a singular concept 

mm 

some requirements are compound statements 

requirements definition minimized when funding short 

wm 

program scope may reduce, but requirements definition completed 

requirements extraction has formal process 

1 X 1 

requirements extraction ad hoc 

change procedures formal 

mm 

change procedures ad hoc 

users/stakeholders somehow involved in requirements definition 

mu 

program team only involved in requirement definition 

management sets requirements for developers 


developers at least partially involved in setting requirements 

requirements changed at least once since baseline established prior to new version 


requirements in baseline has not changed prior to new version or upgrade 

no ranking of requirements 


requirements have priorities assigned 

use-case diagrams (or other models or scenario developments) 

mm 

no models used for requirements extraction 

requirements changes informal 


requirements changes formal 

plan to "freeze" requirements at some designated milestone 


no provision for "freezing" requirements 

requirements must be traceable 


origin of requirements not important 

requirements must be testable 


system developed must be testable 

test plans to determine requirements implemented 


no test plans needed for requirements verification 

requirements have priorities in implementation 


all requirements must be implemented 

some requirements have multiple statements or ideas 


one idea, one statement per requirement 


Requirements Management (page 1 of 2) score 


□□ 
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Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


| ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) \ 

requirements first, then initial development work 


initial development work then requirements 


requirements documentation driving development 


requirements documentation developed in parallel/after development 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


change management procedures used strictly 


change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 


design decisions only after approved requirements stabilized 


requirements summarized w ht w e have developed 


requirements are the blueprint for development 


length of time for requirements work greater than development work 


length of time for requirements w ork less than development w ork 


requirements have design detail 


no design detail in requirements 


requirements creep to be avoided 


requirements creep o.k., but need to be controlled 


freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 


informal change procedure 


change management plan 


no change management plan 


requirements ambiguity always present to some extent 


requirements ambuiguity unacceptable at any level 


testing considered up f rornt during requirements determination 


testing considered dow n the line during development 


requirements development team members different from implementation 


those working on requirements, work on implementation 


start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 


ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWA Y, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 

develop prototype, then determine requirements 

X 

determine requirements prior to any development w ork 


requirements testing done after each iteration 


no testing 

X 

individual changes as necessary 

X 

only block changes made 

X 

development team decides on changes after iteration 

X 

users involved with changes 


changes based on feedback only from user for correction of problems 


changes to upgrade system and correct problems 

X 

funding controls changes and change procedures 

X 

changes control funding 


requirements documentation finalized prior to development 


requirements fluid throughout development (only freeze at end) 

X 

requirements test plans completed prior to development 

X 

requirements test plans completed after development 


requirements first, then initial development work 

X 

initial development work then requirements 


use development effort to learn more about requirements 

X 

define all requirements prior to coding anything 


requirements ambiguity always present to some extent 

X 

requirements ambiguity unacceptable at any level 


requirements have design detail 


no design detail in requirements 

X 

user feedback considered during development 


after development starts, user feedback serves as input to new work 

X 

get something to users as soon as possible for evaluation 

X 

make sure it is complete before releasing 


management dictates requirements 

X 

development team visually represent requirements through rapid prototyping 


new requirements allowed after initial requirements defined 

X 

new requirements not allow ed 



Requirements Management (pg 2 of 2) score [15] +pg 1 score [45] = TOTAL SCORE [60] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 

formal derivation of product metric for estimation of size 

X 

ad hoc size estimation 

ad hoc process evaluation 


formal derivation of at lest one process metric 

develop work breakdown structure (WBS) 

X 

assign work as needs arise 

estimates are developed to fulfill a data call only 


use estimates to plan program 

use estimates to sell program only 


estimates are useful to the project tema for planning purposes 

resource evaluations made for program 

X 

no resource evaluation for planning 

use both bottom up & top down for estimate, use one stakeholders like 


use both bottom up & top down and evaluate significant differences 

estimates made and not updated 


estimates updated throughout program 

resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 

estimations made to fit budget 


budget made from estimations 

estimations compromised to get program 


rather risk loss of program than compromise confident estimations 

cycle time estimations 

X 

no cycle time estimations 

event count estimations 

X 

no event count estimations 

lines of code (LOC) estimation 

X 

no LOC estimation 

function pont (FP) estimation 

X 

no FP estimation 

estimates by algorithmic methods 

X 

estimates by analogy 

expert judgement for estimates 

X 

ad hoc estimates 

estimates by algorithmic methods 

X 

ad hoc estimates 

expert judgement for estimates 


estimates by analogy 

ad hoc estimates 


estimates by analogy 

bottom up estimates 

X 

expert judgement 

top down estimates 

X 

expert judgement 

ad hoc estimates 


any other estimate process 

fuzzy logic estimating method 

X 

no formal estimation methodology 

WBS development from estimates 

X 

WBS development in parallel or prior to estimation completion 

critical path of program determined 

X 

tasks developed but no path is identified 

estimators are program team members 

X 

estimators are outside program team 

management only on estimations 


all team members involved in estimation process 

estimates updated at reviews 

X 

no updates of estimates 

estimates updated at reviews 


estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 

X 

estimate procedures change 

stakeholders are part of estimation process 

X 

stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

X 

important to have WBS as guide, not rigid implementation 

Estimation/Planning Management (page 1 of 2) score 

32 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

X 

estimates for program initiation only 

system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as w hole 

estimates for on-gong resources needed to maintain s/w 

X 

estimates for maintenance not done 

informal re-estimates during development 

X 

formal re-estimates at pre-defined milestones 

formal re-estimates w hen amendment changing the system is introduced 

X 

informal re-estimates w hen amendment changing the system 

person in-charge of estimation walks in a managers office to get an opinion 


meeting(s) organized for purpose of performing cost estimations 

factor analysis prior to commencement of program 

X 

none done 

change control procedures set in place 

X 

no set procedures 

elapsed time and actual work time estimates 

X 

one or the other or neither 

no schedule created 


scheudle created 

schedule not updated 


schedule updated 

schedule follow ed 

X 

schedule not follow ed 

tasks identification arises as program progresses 


detailed level tasks identified prior to program initiation 

scope of program understood by all 

X 

scope not explicitly defined 

quality factors and criteria identified 

X 

no explicit quality factors defined 

no project tracking tools used 


project tracking tools used 

CSCIs identified and tasked 

X 

CSCIs not explicitly identified 

expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 

no cost schedule developed 


cost schedule developed 

no resource schedule developed 


resource schedule developed 

team members, management know at any time if in budget & schedule 

X 

exact budget & schedule status somew hat unclear to at least some 

individual program phases are estimated 

X 

only top level program estimated 

stakeholders/users emphasis understood-quickto field or all complete 

X 

program management sets delivery tradeoffs w ithout outside input 

testing planned with initial program planning 

X 

testing not in initial planning 

documentation not considered ininitial planning 

X 

documentation part of initial planning 

hardw are considered in estimations 

X 

software only considered 

no formal schedule/cost tracking 


formal procedures established for tracking cost and schedule 

earned value set up 

X 

earned value not used 

estimations omit documentation planning 


documentation in estimates 

training omitted in estimates 


training part of estimates 

earned value set up, but not tracked 


earned value tracked 

detailed planning done w ith incomplete set of requirements 

X 

detailed planning done w ith detailed set of requirements 

complete infrastructure support mechanism understood for estimations 

X 

no consideration of infrastructure done for estimations 

team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 

work breakdow n structure (WBS) set up 

X 

no WBS completed 


Estimation/Planning Management (pg 2 of 2) score [32] +pg 1 score [32] = TOTAL SCORE [64] Enter on QMM scoresheet blk b. 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 1 of 2): 
Human Resources 



program team members have clearly deined, segmented roles 


work responsibilities are shared 

formal team building procedures are used 

X 

no formal team building emphasized 

program manager flexible regarding work hours 

X 

program manager maintains strict standards for work hours 

big picture conveyed to all team members by program management 

X 

program management focuses on the partitioned tasks with team 

people issues dealt with primarily through indirect methods (email, memo, etc) 

X 

people issues dealt with primarily through direct methods (face-to-face) 

training is required and planned on a regular basis 

X 

training is ad hoc 

each team member is educated on and understands overall program and their roles 

X 

team members only know their respective areas 

consideration for team members' career goals are reflected in assignments 

X 

team members must adapt to tasks that are assigned 

team members assignments and responsibilities are mostly dictated by PM 


assignments and responsibilities are discussed and agreed upon with PM 

management leads in problem solving 


management facilitates and lets team lead in problem solving 

management welcomes problems as challenges and opportunities 

X 

management views problems as obstacles and grounds for punishment 

team members participate in performance evaluations of peers 

X 

Personnel evaluations are strictly PM responsibility 

management reinforcement feedback sparse and inconsistent, if any 

X 

management provides timely reinforcement feedback for positive behaviors 

management provides basic needs of office facilities fairly well 

X 

office facilities are a drawback to working in the program 

working conditions are fairly comfortable, time off policy fairly good 

X 

working conditions and time off policy is inconsistent and difficult at times 


Communication: 



communications primarily written (email) 

X 

communications primarily verbal (face-to-face) 

detailed instructions: oral presentation, follow-up email 

X 

email only 

formal communication protocol 

X 

informal communications 

external vertical communications restricted 

X 

external vertical communication allowed 

coders notebook weekly accomplishment reports required 

X 

not required 

user-coder relationship established, encouraged, and mediated 

X 

user-coder interaction minimized 

meetings structured to minimize waster time 

X 

meetings unstructured and open ended 

meetings have agenda, objectives, and conclude with action items 

X 

meeting agenda fluid and open ended 

program management and coder communication face to face 

X 

program management and coder communication primarily email 

program team updated regularly regarding organizational & program status 

X 

meetings infrequently scheduled 

open communications is encouraged 

X 

communication hrough chain of command only is encouraged 

program manager accessible for discussions 

X 

program manager difficult to get an appointment to see 

program management (PM) is viewed as separate from team 


PM mixes with team frequently 

management regularly holds team meetings 

X 

meetings are sporadic 

meetings are structured with definite goals and objectives 

X 

meetings are informal 

program management is generally easy to reach and talk to 

X 

PM is usually hard to get a hold of and difficult to talk to 

team-program manager relationship adult-adult 

X 

team-program management relationship parent-child 

schedules are spontaneous and poorly communicated 


schedules must be fixed and rigidly followed and formally reported 

work is seen as complex processes involving team working together 

X 

work broken into pieces with minimal team member interaction 

action items often is poorly disseminated and usually not followed through 


action items communicated and followed through thoroughly 

team members require frequent clarifications by PM for assigned tasks 


team members rarly require clarifications by PM for assigned tasks 



83 


XX X X XX 


















































































































Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

lead through personal attention to others 

run as much of the organization as possible 

direct and domineering style 

traditional leaders respect hierarchy 

w in cooperation rather than demand it 

act strongly and forcefully in the field of ideas 

consults w ith team members to find solutions to problems 

keep people w ell informed 

make things happen by focusing on the immediate problems 
manage others loosely and prefer minimal supervision 
leadership, management decisions exclusively by program management 
team-program manager relationship adult-adult 
program management makes decisions but gets inputs from team 
w hen a problem arises: management takes over to solve it 
leadership is do as I say, not do as I do 
program expectation not influenced by PM 

PM gives freedom to team, but has no mentoring for members (abdication) 

promgram management waits and sees what happens then plans 

program management is constantly reacting to emergencies 

facilitative approach to solving problems 

program management is complex, takes much time to understand 

program management prefers to plunge right in 

program management reacts spur of the moment 


X short tern program and immediate w ork focus 
X action-oriented leadership approach 


let team make decisions as much as possible X 


encourage independence in others X 


do w hat needs to be done X 


X I tough-minded with others 


prefer to lead other independent types while seeking auto no my for self X 


X consults team members to get validation of PMs predetermined solutions 
X only as much know ledge as necessary for their w ork 
X long range focus and de-emphasize current problems 
X follow traditional procedures and rules conscientiously 

program management makes decisions but gets inputs from team X 

X team-program management relationship parent-child 

all program team members responsible for program decisions 
management lets the team solve the problems 
X leadership by example 
X program expectation managed by FM 

FM empowers teams by mentoring members to be leaders 
X management plans far in advance 

management is one step ahead of problems 
X take charge readily and often 
X management is simple, easy to figure out 

takes time to separate things to be done and order of doing them 
methodically follows plans 



Technical Competency of the Program Manager: 

PM has technical experience particular to the particular s/w program X PM relies on team members solely 

PM participates in technical reviews X PM only in non-technical reviews 

PM participates in making technical decisions w hen problems arise X PM delegates technical questions 

PM does not get involved discussing technical options PM contributes to technical options being discussed X 

PM does not review technical options and decisions PM review s technical options and decisions X 

PM actively attempts to keep up-to-date with current techno logy and standards PM is removed from cutting edge technology issues X 

PM receives technical periodicals and occasionallyreferences applicable articles X PM doesn't read periodicals nor reference current articles to team 

PM doesn't have technical background (or education) X PM has technical background (or education) 

team members avoid PM w hen they need technical advice team members generally consider talking to PM regarding technical issues X 


HR [13] + Comm. [21] + Leadership [20] + Tech. Competency [8] = People Mgmt. score [60] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

X 

RM is informal, if at all 


a risk management plan exists 

X 

no risk management plan is developed 


RM is more of a data call than a useful document 


RM drives decisions on the program 

X 

RM is done prior to the program beginning 


RM is done prior and during program execution 

X 

RM is only done during the program execution 


RM is done prior and during program execution 

X 

risks are generalized through the whole program 


risks are categorized 

X 

risk management is done internally, only 

X 

an outside organization also contributes to the RM process 


risk is a management function 

X 

risk is a program team function 


risks are precisely articulated 

X 

risks are generalized, if at all 


each risk has a consequence 

X 

consequences are generalized, if at all 


a mitigation strategy is completed for each risk 

X 

mitigation strategy is generalized, if at all 


contingency plans are developed for a RM plan 

X 

contingency plans are ad hoc as problems arise in the program 


risks are anticipated 

X 

if problems arise, management will deal with it 


the program doesn't have any risk 


programs that do not have risk, have problems 

X 

risk management is automated 

X 

risk management may use tools, but depend on human input 


risks are assigned probabilities 

X 

probabilities are not relevant for RM 


all risks are potential problems, relative priorities for risks are not useful 

X 

risks are weighed relative to other program risks and thus prioritized 


risk management information is only shared internally 

X 

risk management information is shared with all stakeholders 


risk analysis uses ordinal rankings 

X 

risk analysis uses actual measurements with a mathematical model 


regret analysis used 

X 

no regret analysis done 


attach probabilities to future events 

X 

no probabilities associated with future events 


assessing risks with mechanical meethods 

X 

risks should be compared to other risks and sorted 

X 

risk status tracked 

X 

not tracked 


technical risks examined 

X 

no technical risks examined 


process risks examined 

X 

no process risks examined 


product risks examined 

X 

no product risks examined 


stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 


checklists used to identify risks 

X 

no checklists used 


risks are tracked 

X 

no tracking or monitoring of risks 


each risk has an impact 

X 

no impact analysis of risk 


each risk has a mitigation plan 

X 

no individual risk mitigation 


risks monitored by priority 

X 

no special attention to track higher priority risks 


risk assessment is formalized 

X 

no formal risk assessment 


risk control is formalized 

X 

no formal risk control 


integration risks not considered 


integration risks examined 

X 

Risk Management (page 1 of 2) score 

30 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 

unforeseen risks have occurred in program 


any risk that came up had been identified previously 

personnel risks examined 

X 

no personnel risks examined 

estimation risks examined 

X 

no estimation risks examined 

planning risks examined 

X 

no planning risks examined 

requirements risks examined 

X 

no requirements risks examined 

resource risks examined 

X 

no resource risks examined 

risk management plan updated regularly 

X 

no regular risk management plan updates 

risks charted 

X 

risks not charted 

performance risks examined 

X 

performance risks not examined 

program management self risks examined 

X 

no program management risks examined 

risk from program constraints examined 


no program constraint risks examined 

each category of risks are prioritized 

X 

no prioritization 

each category of risks are evaluated for impact 

X 

no impact analysis performed 

each category of risks have control strategy 

X 

no control strategy 

documentation risks examined 

X 

no documentation risks examined 

regret matrix tracked 

X 

no regret matrix or not tracked 

communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 

taxonomy-based questionnaire used to identify risks 


taxonomy-based questionnaire not used 

associated hardware risks examined 

X 

no consideration for hardware risks 

integration risks examined 

X 

integration risks not examined 

communication risks examined 

X 

communication risks not examined 

leadership risks examined 


leadership risks not considered 

risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 

risk documentation forms used 

X 

no risk documentation forms used 

dependency risks examined 

X 

no dependency risks examined 

alternatives like risk avoidance considered for high risk items 

X 

no consideration of risk avoidance 

documented risk statements use a condition-consequence type format 

X 

condition-consequence of risk statements not clearly defined 

no assignment of ownership of risk mitigation action 


each risk mitigation action is assigned to an individual for resolution 

calculation of risk exposure made (probability X loss, for each risk) 


no risk exposure calculations 

oral communication of risks only 


risks written in a way that communicates nature and status of factors 

triggers used to quantify risk conditions present 

X 

risk conditions present are all subjective 

risk "czar" in program for monitoring risks 


no special positions/responsibilities for risk monitoring 

post-program review completed (scheduled) for unanticipated problems ID 


no post-program reviews completed or scheduled 

no schedule risks examined 


risks to schedule investigated 


Risk Management (pg 2 of 2) score [30] +pg 1 score [31] = TOTAL SCORE [61] Enter on QMM scoresheet blk d. 
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E. PROGRAM C - PROGRAM MANAGER 


1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

46 

e 

7 

53 


0.92 

= 

48.76 

Est./Planning 

Management 

D 

60 

D 

44 

104 


0.67 

= 

69.68 

People 

Management 

c 

18 

g 

-15 

3 


1.86 

= 

5.58 

Risk Management 

D 

62 

D 

47 

109 


0.55 

= 

59.95 


QMM SCORE 183.97 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 

QMM percentage score: 47.78% 


Objective/Subjective view of the overall success of program A on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Program Manager 

Success Score: 6 
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Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e Yes No N/A 


1 

PM chose to have a formal requirements list 



X 

2 

Requirements recorded in some way 



X 

3 

Written requirements were part of some formal document 



X 

4 

Written requirements were informal 



X 

5 

At least some requirements were oral only 



X 

6 

All stakeholders were identified 



X 

7 

All stakeholders participated in the requirements extraction 

X 



8 

Some stakeholders participated in the requirements extraction 



X 

9 

Management extracted requirements, no stakeholder involvement 


X 


10 

Management passed requirements to development team 



X 

11 

Stakeholders not involvved in Management extraction, but approved 



X 

12 

Management gets inputs from stakeholders, then develops requirements 


~ir 


13 

Developers work informally with users to arrive at requirements 



X 

14 

Same as 13, but management oversees and formalizes 



X 

If a waterfall or sequential development strategy: | 

15 

All requirements complete before design 




16 

Some requirements left incomplete prior to design 




17 

Requirements informal prior to design effort 




18 

Requirements serve as input 




19 

Length of time for requirements w ork greater than development w ork 




20 

Requirements developed in parallel to design 




OR If a prototype, throwaway, or other development strategy: | 

15 

Learn about requirements through development efforts 

X 



16 

No coding until all requirements are defined 



X 

17 

Requirements formal prior to design effort 



X 

18 

Requirements serve as output 



X 

19 

Requirements definition work in parallel to development efforts 



X 

20 

Requirements developed in parallel to design 

"IT 



21 

Are requirements frozen at some phase 



X 

22 

Change management exists 



X 

23 

Change management is formal 


X 


24 

Project strategy is consistent throughout development 



X 

25 

Requirements are updated 



X 

26 

Configuration Management (CM) exists 

~ 



27 

CM is formal 



X 

28 

Requirements are testable 



X 

29 

Requirements testing considered/implemented during extraction 



X 

30 

Requirements testing plan exists 



X 

31 

Requirements testing is formal 



X 

32 

All requirements have priorities 



X 

33 

All requirements must be implemented 



X 

34 

Requirements are tested 



X 

35 

All requirements are equally important 



X 

36 

At least some requirements have priorities 



X 

37 

All requirements are traceable 



X 

38 

Traceability not important 



X 

39 

Each requirement has an author 



X 

40 

Who authored requirement is not important 



X 

41 

Initial set of requirements to be implemented, no requirements creep 



X 

42 

Structured and tracked changes to requirements only 



X 

43 

Change is inevitable, changes allow ed at all times 



X 

44 

Change is inevitable, but changes limited 



X 

45 

Requirements control funding 



X 

46 

Requirements history kept 



X 

47 

Baseline established for requirements at some point prior to develop 




TOTAL SCORING 

B 

-Q- 
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3. Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f 

Yes 

No 

N/A 

1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 

X 



4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 


X 


5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 



~ 

7 

Time measure process metric used (cycle time) 

X 



8 

Process metrics tracked and updated throughout program execution 

X 



9 

Program cost estimations made from product or process metrics 

X 



10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 

X 



12 

Program's primary purpose, including major functions and deliverables known 



~>r 

13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 

X 



15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 

X 



18 

Quality assurance plan or similar to aid in detecting defects early in program 

X 



19 

COCOMO estimates performed 

X 



20 

CSCI clearlydefined and tasked 

X 



21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 


X 


29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 


X 


33 

Automated program tracking used 


X 


34 

PM usually thorough in tracking and reporting schedules and financials 

X 



35 

WBS developed only as data call 


X 


36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 



“x" 

38 

Estimations are done using both top down and bottoms up approaches 



X 

39 

All program team members involved in planning process 



X 

40 

Hardware also considered in estimaation process 

X 



41 

Program history compiled 

X 



42 

System upgrades (SCR) software changes requests estimated individually 

X 



43 

Management duties apart of each team member's responsibilities 



“x - 

44 

PM dictates schedules to program team 

X 



45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 



~ 

47 

Test planning done at the start of the program 



X 

48 

Estimations are completed bythose performing the tasks 

X 



49 

Sensitivity analysis performed for program choices 

X 


~ 

50 

Software deployment planning completed 

X 


X 

TOTAL SCORING 

ESI 
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4. People Management Questionnaire Responses 


No. People Management Questionnaire - Total: Block g 

Yes 

No 

N/A 

1 

PM is accessible in person by each team member 


X 


2 

PM is accessible via email (memo, letter) by each team member 


X 


3 

PM is accessible via phone by each team member 


X 


4 

PM not only considers a person's suitability, not also desire to be on a team 


X 


5 

PM consults with each team member regarding their career goals 


X 


6 

PM regularly holds meetings to inform team of program progress 


X 


7 

PM solicits opinions from team members before making decisions 


X 


8 

PM lets teams make decisions affecting their work 


X 


9 

PM freuently makes decisions without any consultation with members 



X 

10 

PM understands the technology/language of the program 


X 


11 

PM is able to communicate with other the technical issues in the program 


X 


12 

PM prioritized problems or conflicts within the program 


X 


13 

PM assists team members in developing/advising of career path 


X 


14 

PM empowers program members to recommend hiring new team members 


X 


15 

PM empowers program members to recommend firings of other members 



X 

16 

PM specifically assigns work to each program member 



X 

17 

PM sets communication protocol 



X 

18 

PM allows unrestricted communications 



X 

19 

PM encourages or requires training for each individual 


X 


20 

PM takes control in difficult/roblem areas 


X 


21 

PM looks ahead to new programs, new upgrades of existing program 


X 


22 

PM maintains regular communications with all stakeholders 



X 

23 

PM maintains regular communications with users 


X 


24 

PM encourages program team communication with users 


X 


25 

PM encourages program team communication with stakeholders 



X 

26 

PMfacilitates horizontal communication within program 



X 

27 

PM facilitates communication during integration 



X 

28 

PM holds meetings without clear objectives 

X 



29 

PM must approve all decisions within the program 

X 



30 

PM must approve all interactions with stakeholders 

X 



31 

PM must approve all interactions with users 

X 



32 

PM makes all presentations to stakeholders/users 

X 



33 

PM is considered "flexible" in terms of program members personal issues 


X 


34 

PM, at least occasionally, schedules/promotes outside work team activities 


X 


35 

PM is readily willing to listen to program prblems and complaints 



X 

36 

PM takes action to resolve program problems and complaints 



X 

37 

PM is generally respected by stakeholders, users, and organization 



X 

38 

PM sometimes fails to grasp important technical issues in program 



X 

39 

PM recruits program team members from outside organization 


X 


40 

PM participates in technical reviews 



X 

41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual’s tasks are specific, each exposed to the "bigger picture" 



X 

43 

PM has clearly defined his/her expectations for each individual 



X 

44 

PMdelegation of duties is usuallyseemless in execution 


X 


45 

PM acts as facilitator to solving personnel conflicts 


X 


46 

PM attempts to motivate individuals on the program team 


X 


47 

PM clearly spearates technical from managerial roles for individuals 

X 



48 

PM directs how he/she expects the task to be accomplished 

X 



49 

PM directs what needs to be done, but does not direct how 

X 



50 

PM attempts to spotlight individuals in the program for positive exposure 



X 

TOTAL SCORING 

-4 

-11 
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5. Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Total: Block h 

Yes 

No 

N/A 

1 

Risk Management (RM) is specifically an activity in the program 

X 



2 

RMis formal and documented 

X 



3 

Aspecific RM Ian exists 

X 



4 

RMis required in the program, but not used during the program 

X 



5 

RM is done prior to the program execution 


X 


6 

RM is done by an outside entity to the development 

X 



7 

RMis done internally only 

X 



8 

RMis both internally performed and externally assessed 


X 


9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is oniya management function 

X 



11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 


X 


13 

Risks are only generalized 

X 



14 

Each risk is delineated 

X 



15 

Each risk has a consequence 

X 



16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 

X 



18 

Risk Management is automated 

X 



19 

Risks are tracked 

X 



20 





21 

Regret analysis performed 


X 


22 

RM drives decisions in the program 



X 

23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 


X 


25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 

X 



28 

Risk Assessment done prior to program start 

X 



29 

Risk Assessment includes personnal risk 

X 



30 

RM uses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 

X 



34 

Risk Assessment is briefed organization structure above program manager 

X 



35 

Risk Assessment includes requirements risks 

X 



36 

Risk Assessment includes user risks (too little involvement of user) 

X 



37 

Risk Assessment includes documentation risks 

X 



38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 

X 



42 

Documentation proof exists to demonstrate following risk management plan 

X 



43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 

X 



45 

Other organizational checklists used for risk assessment 

X 



46 

Internal organizational checklists used for risk assessment 

X 



47 

Risk Assessment information contributed to internal or other database 

X 



48 

Risk Assessment includes internal organization risks 

X 



49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 

m 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

X 

informal requirement list 


w ritten requirements 

X 

oral requirements 


requirements informal, but recorded 

X 

requirements not recorded 


requirements as part of an SRS (or other formal repository) 

X 

requirements informally recorded 


requirements taken as is from customer 


look to reformulate, interview in-depth, or otherwise re-validate 

X 

only one development strategy used 

X 

strategies not consistent, used at different times 


stakeholders as part of requirements development 

X 

stakeholders approving requirements after formulated by development team 


requirements are testable 

X 

requirements have no test plans 


informal test plan or no test plan 


formal test plan 

X 

test team involved w ith requirements 

X 

no test team input or plans during requirements development 


only a percentage of requirements present in baseline 


baseline must contain all requirements 

X 

requirements documentation has hierarchical structure 

X 

all requirements must be implemented 


requirements have listed responsible party 

X 

requirements origin not important 


requirements documentation have versions 

X 

no requirements history 


requirements have specific attribute values 

X 

requirements all rank evenly 


funding controls requirements definition 


requirements definition controls funding 

X 

reqquirements are top dow n 

X 

requirements are bottom up 


users/stakeholders are identified and interviewed (market survey) 


no special consideration to identify users/stakeholders 

X 

each requirement has a singular concept 


some requirements are compound statements 

X 

requirements definition minimized when funding short 

X 

program scope may reduce, but requirements definition completed 


requirements extraction has formal process 

X 

requirements extraction ad hoc 


change procedures formal 

X 

change procedures ad hoc 


users/stakeholders somehow involved in requirements definition 

X 

program team only involved in requirement definition 


management sets requirements for developers 


developers at least partially involved in setting requirements 

X 

requirements changed at least once since baseline established prior to new version 

X 

requirements in baseline has not changed prior to new version or upgrade 


no ranking of requirements 


requirements have priorities assigned 

X 

use-case diagrams (or other models or scenario developments) 

X 

no models used for requirements extraction 


requirements changes informal 


requirements changes formal 

X 

plan to "freeze" requirements at some designated milestone 


no provision for "freezing" requirements 

X 

requirements must be traceable 

X 

origin of requirements not important 


requirements must be testable 


system developed must be testable 

X 

test plans to determine requirements implemented 


no test plans needed for requirements verification 

X 

requirements have priorities in implementation 

X 

all requirements must be implemented 


some requirements have multiple statements or ideas 

X 

one idea, one statement per requirement 



Requirements Management (page 1 of 2) score | 29 
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Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


| ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) \ 

requirements first, then initial development work 


initial development work then requirements 


requirements documentation driving development 


requirements documentation developed in paraliel/after development 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


change management procedures used strictly 


change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 


design decisions only after approved requirements stabilized 


requirements summarized w ht w e have developed 


requirements are the blueprint for development 


length of time for requirements work greater than development work 


length of time for requirements w ork less than development w ork 


requirements have design detail 


no design detail in requirements 


requirements creep to be avoided 


requirements creep o.k., but need to be controlled 


freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 


informal change procedure 


change management plan 


no change management plan 


requirements ambiguity always present to some extent 


requirements ambuiguity unacceptable at any level 


testing considered up f rornt during requirements determination 


testing considered dow n the line during development 


requirements development team members different from implementation 


those working on requirements, work on implementation 


start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 


ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWA Y, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 

develop prototype, then determine requirements 

X 

determine requirements prior to any development w ork 


requirements testing done after each iteration 

X 

no testing 


individual changes as necessary 

X 

only block changes made 


development team decides on changes after iteration 


users involved with changes 

X 

changes based on feedback only from user for correction of problems 


changes to upgrade system and correct problems 

X 

funding controls changes and change procedures 

X 

changes control funding 


requirements documentation finalized prior to development 


requirements fluid throughout development (only freeze at end) 

X 

requirements test plans completed prior to development 

X 

requirements test plans completed after development 


requirements first, then initial development work 

X 

initial development w ork then requirements 


use development effort to learn more about requirements 

X 

define all requirements prior to coding anything 


requirements ambiguity always present to some extent 

X 

requirements ambiguity unacceptable at any level 


requirements have design detail 

X 

no design detail in requirements 


user feedback considered during development 

X 

after development starts, user feedback serves as input to new work 


get something to users as soon as possible for evaluation 

X 

make sure it is complete before releasing 


management dictates requirements 

X 

development team visually represent requirements through rapid prototyping 


new requirements allowed after initial requirements defined 

X 

new requirements not allowed 



Requirements Management (pg 2 of 2) score [17] +pg 1 score [29] = TOTAL SCORE [46] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 

formal derivation of product metric for estimation of size 

X 

ad hoc size estimation 

ad hoc process evaluation 


formal derivation of at lest one process metric 

develop work breakdown structure (WBS) 

X 

assign work as needs arise 

estimates are developed to fulfill a data call only 


use estimates to plan program 

use estimates to sell program only 


estimates are useful to the project tema for planning purposes 

resource evaluations made for program 

X 

no resource evaluation for planning 

use both bottom up & top down for estimate, use one stakeholders like 


use both bottom up & top down and evaluate significant differences 

estimates made and not updated 


estimates updated throughout program 

resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 

estimations made to fit budget 


budget made from estimations 

estimations compromised to get program 


rather risk loss of program than compromise confident estimations 

cycle time estimations 

X 

no cycle time estimations 

event count estimations 

X 

no event count estimations 

lines of code (LOC) estimation 

X 

no LOC estimation 

function pont (FP) estimation 

X 

no FP estimation 

estimates by algorithmic methods 

X 

estimates by analogy 

expert judgement for estimates 

X 

ad hoc estimates 

estimates by algorithmic methods 

X 

ad hoc estimates 

expert judgement for estimates 

X 

estimates by analogy 

ad hoc estimates 


estimates by analogy 

bottom up estimates 

X 

expert judgement 

top down estimates 

X 

expert judgement 

ad hoc estimates 


any other estimate process 

fuzzy logic estimating method 

X 

no formal estimation methodology 

WBS development from estimates 

X 

WBS development in parallel or prior to estimation completion 

critical path of program determined 

X 

tasks developed but no path is identified 

estimators are program team members 

X 

estimators are outside program team 

management only on estimations 

X 

all team members involved in estimation process 

estimates updated at reviews 

X 

no updates of estimates 

estimates updated at reviews 

X 

estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 

X 

estimate procedures change 

stakeholders are part of estimation process 

X 

stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

X 

important to have WBS as guide, not rigid implementation 


Estimation/Planning Management (page 1 of 2) score | 33 | 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

X 

estimates for program initiation only 

system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as w hole 

estimates for on-gong resources needed to maintain s/w 

X 

estimates for maintenance not done 

informal re-estimates during development 


formal re-estimates at pre-defined milestones 

formal re-estimates w hen amendment changing the system is introduced 

X 

informal re-estimates w hen amendment changing the system 

person in-charge of estimation walks in a managers office to get an opinion 

X 

meeting(s) organized for purpose of performing cost estimations 

factor analysis prior to commencement of program 


none done 

change control procedures set in place 

X 

no set procedures 

elapsed time and actual work time estimates 

X 

one or the other or neither 

no schedule created 


scheudle created 

schedule not updated 


schedule updated 

schedule followed 

X 

schedule not followed 

tasks identification arises as program progresses 

X 

detailed level tasks identified prior to program initiation 

scope of program understood by all 

X 

scope not explicitly defined 

quality factors and criteria identified 


no explicit quality factors defined 

no project tracking tools used 


project tracking tools used 

CSCIs identified and tasked 


CSCIs not explicitly identified 

expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 

no cost schedule developed 


cost schedule developed 

no resource schedule developed 


resource schedule developed 

team members, management know at any time if in budget & schedule 

X 

exact budget & schedule status somew hat unclear to at least some 

individual program phases are estimated 

X 

only top level program estimated 

stakeholders/users emphasis understood-quickto field or all complete 

X 

program management sets delivery tradeoffs w ithout outside input 

testing planned w ith initial program planning 

X 

testing not in initial planning 

documentation not considered ininitial planning 


documentation part of initial planning 

hardw are considered in estimations 


software only considered 

no formal schedule/cost tracking 


formal procedures established for tracking cost and schedule 

earned value set up 

X 

earned value not used 

estimations omit documentation planning 


documentation in estimates 

training omitted in estimates 


training part of estimates 

earned value set up, but not tracked 


earned value tracked 

detailed planning done w ith incomplete set of requirements 


detailed planning done w ith detailed set of requirements 

complete infrastructure support mechanism understood for estimations 

X 

no consideration of infrastructure done for estimations 

team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 

w ork breakdow n structure (WBS) set up 

X 

no WBS completed 


Estimation/Planning Management (pg 2 of 2) score [27] +pg 1 score [33] = TOTAL SCORE [60] Enter on QMM scoresheet blk b. 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 1 of 2): 
Human Resources 


program team members have clearly deined, segmented roles 

X 

work responsibilities are shared 


formal team building procedures are used 


no formal team building emphasized 

X 

program manager flexible regarding work hours 


program manager maintains strict standards for work hours 

X 

big picture conveyed to all team members by program management 


program management focuses on the partitioned tasks with team 

X 

people issues dealt with primarily through indirect methods (email, memo, etc) 


people issues dealt with primarily through direct methods (face-to-face) 

X 

training is required and planned on a regular basis 

X 

training is ad hoc 


each team member is educated on and understands overall program and their roles 


team members only know their respective areas 

X 

consideration for team members' career goals are reflected in assignments 


team members must adapt to tasks that are assigned 

X 

team members assignments and responsibilities are mostly dictated by PM 


assignments and responsibilities are discussed and agreed upon with PM 

X 

management leads in problem solving 

X 

management facilitates and lets team lead in problem solving 


management welcomes problems as challenges and opportunities 


management views problems as obstacles and grounds for punishment 

X 

team members participate in performance evaluations of peers 


Personnel evaluations are strictly PM responsibility 

X 

management reinforcement feedback sparse and inconsistent, if any 

X 

management provides timely reinforcement feedback for positive behaviors 


management provides basic needs of office facilities fairly well 


office facilities are a drawback to working in the program 

X 

working conditions are fairly comfortable, time off policy fairly good 


working conditions and time off policy is inconsistent and difficult at times 

X 


Communication: 


communications primarily written (email) 

X 

communications primarily verbal (face-to-face) 


detailed instructions: oral presentation, follow-up email 


email only 

X 

formal communication protocol 


informal communications 

X 

external vertical communications restricted 

X 

external vertical communication allowed 


coders notebook weekly accomplishment reports required 


not required 

X 

user-coder relationship established, encouraged, and mediated 


user-coder interaction minimized 

X 

meetings structured to minimize waster time 


meetings unstructured and open ended 

X 

meetings have agenda, objectives, and conclude with action items 


meeting agenda fluid and open ended 

X 

program management and coder communication face to face 


program management and coder communication primarily email 

X 

program team updated regularly regarding organizational & program status 


meetings infrequently scheduled 

X 

open communications is encouraged 


communication hrough chain of command only is encouraged 

X 

program manager accessible for discussions 


program manager difficult to get an appointment to see 

X 

program management (PM) is viewed as separate from team 

X 

PM mixes with team frequently 


management regularly holds team meetings 

X 

meetings are sporadic 


meetings are structured with definite goals and objectives 

X 

meetings are informal 


program management is generally easy to reach and talk to 


PM is usually hard to get a hold of and difficult to talk to 

X 

team-program manager relationship adult-aduit 


team-program management relationship parent-child 

X 

schedules are spontaneous and poorly communicated 


schedules must be fixed and rigidly followed and formally reported 

X 

work is seen as complex processes involving team working together 


work broken into pieces with minimal team member interaction 

X 

action items often is poorly disseminated and usually not followed through 

X 

action items communicated and followed through thoroughly 


team members require frequent clarifications by PM for assigned tasks 

X 

team members rarly require clarifications by PM for assigned tasks 



















































































































Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

X 

short tern program and immediate work focus 

X 

X 

X 

X 

X 

X 

lead through personal attention to others 

X 

action-oriented leadership approach 

run as much of the organization as possible 

X 

let team make decisions as much as possible 

direct and domineering style 

X 

encourage independence in others 

traditional leaders respect hierarchy 

X 

do what needs to be done 

win cooperation rather than demand it 


tough-minded with others 

act strongly and forcefully in the field of ideas 

X 

prefer to lead other independent types while seeking autonomy for self 

consults with team members to find solutions to problems 

X 

consults team members to get validation of PM's predetermined solutions 

keep people well informed 


only as much knowledge as necessary for their work 

make things happen by focusing on the immediate problems 

X 

long range focus and de-emphasize current problems 

manage others loosely and prefer minimal supervision 


follow traditional procedures and rules conscientiously 

leadership, management decisions exclusively by program management 

X 

program management makes decisions but gets inputs from team 

team-program manager relationship adult-adult 


team-program management relationship parent-child 

program management makes decisions but gets inputs from team 

X 

all program team members responsible for program decisions 

when a problem arises: management takes over to solve it 

X 

management lets the team solve the problems 

leadership is do as 1 say, not do as 1 do 

X 

leadership by example 

program expectation not influenced by PM 

X 

program expectation managed by PM 

PM gives freedom to team, but has no mentoring for members (abdication) 

X 

PM empowers teams by mentoring members to be leaders 

promgram management waits and sees what happens then plans 

X 

management plans far in advance 

program management is constantly reacting to emergencies 

X 

management is one step ahead of problems 

facilitative approach to solving problems 

X 

take charge readily and often 

program management is complex, takes much time to understand 


management is simple, easy to figure out 

program management prefers to plunge right in 

X 

takes time to separate things to be done and order of doing them 

program management reacts spur of the moment 

X 

methodically follows plans 

Technical Competency of the Program Manager: 

PM has technical experience particular to the particular s/w program 

X 

PM relies on team members solely 

X 

X 

X 

X 

X 

PM participates in technical reviews 

X 

PM only in non-technical reviews 

PM participates in making technical decisions when problems arise 

X 

PM delegates technical questions 

PM does not get involved discussing technical options 

X 

PM contributes to technical options being discussed 

PM does not review technical options and decisions 

X 

PM reviews technical options and decisions 

PM actively attempts to keep up-to-date with current technology and standards 


PM is removed from cutting edge technology issues 

PM receives technical periodicals and occasionally references applicable articles 


PM doesn't read periodicals nor reference current articles to team 

PM doesn't have technical background (or education) 


PM has technical background (or education) 

team members avoid PM when they need technical advice 


team members generally consider talking to PM regarding technical issues 


HR [2] + Comm. [4] + Leadership [6] + Tech. Competency [6] = People Mgmt. score [18] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

X 

RM is informal, if at all 

a risk management plan exists 

X 

no risk management plan is developed 

RM is more of a data call than a useful document 


RM drives decisions on the program 

RM is done prior to the program beginning 


RM is done prior and during program execution 

RM is only done during the program execution 


RM is done prior and during program execution 

risks are generalized through the whole program 

X 

risks are categorized 

risk management is done internally, only 


an outside organization also contributes to the RM process 

risk is a management function 


risk is a program team function 

risks are precisely articulated 

X 

risks are generalized, if at all 

each risk has a consequence 

X 

consequences are generalized, if at all 

a mitigation strategy is completed for each risk 

X 

mitigation strategy is generalized, if at all 

contingency plans are developed for a RM plan 

X 

contingency plans are ad hoc as problems arise in the program 

risks are anticipated 


if problems arise, management will deal with it 

the program doesn't have any risk 


programs that do not have risk, have problems 

risk management is automated 


risk management may use tools, but depend on human input 

risks are assigned probabilities 

X 

probabilities are not relevant for RM 

all risks are potential problems, relative priorities for risks are not useful 


risks are weighed relative to other program risks and thus prioritized 

risk management information is only shared internally 


risk management information is shared with all stakeholders 

risk analysis uses ordinal rankings 


risk analysis uses actual measurements with a mathematical model 

regret analysis used 

X 

no regret analysis done 

attach probabilities to future events 

X 

no probabilities associated with future events 

assessing risks with mechanical meethods 


risks should be compared to other risks and sorted 

risk status tracked 

X 

not tracked 

technical risks examined 

X 

no technical risks examined 

process risks examined 

X 

no process risks examined 

product risks examined 

X 

no product risks examined 

stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 

checklists used to identify risks 

X 

no checklists used 

risks are tracked 

X 

no tracking or monitoring of risks 

each risk has an impact 

X 

no impact analysis of risk 

each risk has a mitigation plan 

X 

no individual risk mitigation 

risks monitored by priority 

X 

no special attention to track higher priority risks 

risk assessment is formalized 

X 

no formal risk assessment 

risk control is formalized 

X 

no formal risk control 

integration risks not considered 


integration risks examined 


Risk Management (page 1 of 2) score | 33 | 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 

unforeseen risks have occurred in program 


any risk that came up had been identified previously 

personnel risks examined 

X 

no personnel risks examined 

estimation risks examined 

X 

no estimation risks examined 

planning risks examined 

X 

no planning risks examined 

requirements risks examined 

X 

no requirements risks examined 

resource risks examined 

X 

no resource risks examined 

risk management plan updated regularly 

X 

no regular risk management plan updates 

risks charted 

X 

risks not charted 

performance risks examined 

X 

performance risks not examined 

program management self risks examined 

X 

no program management risks examined 

risk from program constraints examined 

X 

no program constraint risks examined 

each category of risks are prioritized 

X 

no prioritization 

each category of risks are evaluated for impact 

X 

no impact analysis performed 

each category of risks have control strategy 

X 

no control strategy 

documentation risks examined 

X 

no documentation risks examined 

regret matrix tracked 

X 

no regret matrix or not tracked 

communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 

taxonomy-based questionnaire used to identify risks 

X 

taxonomy-based questionnaire not used 

associated hardware risks examined 

X 

no consideration for hardware risks 

integration risks examined 

X 

integration risks not examined 

communication risks examined 

X 

communication risks not examined 

leadership risks examined 

X 

leadership risks not considered 

risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 

risk documentation forms used 

X 

no risk documentation forms used 

dependency risks examined 

X 

no dependency risks examined 

alternatives like risk avoidance considered for high risk items 

X 

no consideration of risk avoidance 

documented risk statements use a condition-consequence type format 

X 

condition-consequence of risk statements not clearly defined 

no assignment of ownership of risk mitigation action 

X 

each risk mitigation action is assigned to an individual for resolution 

calculation of risk exposure made (probability X loss, for each risk) 


no risk exposure calculations 

oral communication of risks only 

X 

risks written in a way that communicates nature and status of factors 

triggers used to quantify risk conditions present 


risk conditions present are all subjective 

risk "czar" in program for monitoring risks 


no special positions/responsibilities for risk monitoring 

post-program review completed (scheduled) for unanticipated problems ID 


no post-program reviews completed or scheduled 

no schedule risks examined 


risks to schedule investigated 


Risk Management (pg 2 of 2) score [29] +pg 1 score [33] = TOTAL SCORE [62] Enter on QMM scoresheet blk d. 
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F. PROGRAM C - ASSOCIATE 

1. QMM Summary Score Sheet 


QMM Scoresheet 

Part One 

Part Two 

Total 


Importance 


Weighted 

Category 

Score 

Score 

Score 


Coefficient 


Score 

Requirements 

Management 

a 

45 

e 

5 

50 


0.92 

= 

46 

Est./Planning 

Management 

D 

54 

D 

38 

92 


0.67 

= 

61.64 

People 

Management 

c 

16 

g 

-15 

1 


1.86 

= 

1.86 

Risk Management 

D 

61 

D 

46 

107 


0.55 

= 

58.85 


QMM SCORE 168.35 


Max. QMM score possible 528.00 

Min. QMM score possible -130.86 

QMM percentage score: 45.41% 


Objective/Subjective view of the overall success of program A on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Survey Participant: Associate 

Success Score: 6 


100 











































2 , 


Requirements Management Questionnaire Responses 


No. Requirements Management Questionnaire - Total: Block e_Yes No N/A 


1 

PM chose to have a formal requirements list 


X 


2 

Requirements recorded in some way 



X 

3 

Written requirements were part of some formal document 



X 

4 

Written requirements were informal 



X 

5 

At least some requirements were oral only 



X 

6 

All stakeholders were identified 



X 

7 

All stakeholders participated in the requirements extraction 



X 

8 

Some stakeholders participated in the requirements extraction 



X 

9 

Management extracted requirements, no stakeholder involvement 

X 



10 

Management passed requirements to development team 


X 


11 

Stakeholders not involvved in Management extraction, but approved 


X 


12 

Management gets inputs from stakeholders, then develops requirements 


X 


13 

Developers work informally with users to arrive at requirements 


X 


14 

Same as 13, but management oversees and formalizes 


X 


If a waterfall or sequential development strategy: | 

15 

All requirements complete before design 




16 

Some requirements left incomplete prior to design 




17 

Requirements informal prior to design effort 




18 

Requirements serve as input 




19 

Length of time for requirements work greater than development work 




20 

Requirements developed in parallel to design 




OR If a prototype, throwaway, or other development strategy: j 

15 

Learn about requirements through development efforts 



X 

16 

No coding until all requirements are defined 



X 

17 

Requirements formal prior to design effort 



X 

18 

Requirements serve as output 



X 

19 

Requirements definition work in parallel to development efforts 

X 



20 

Requirements developed in parallel to design 



X 

21 

Are requirements frozen at some phase 



X 

22 

Change management exists 



X 

23 

Change management is formal 



X 

24 

Project strategy is consistent throughout development 


X 


25 

Requirements are updated 


X 


26 

Configuration Management (CM) exists 


X 


27 

CM is formal 

X 



28 

Requirements are testable 



X 

29 

Requirements testing considered/implemented during extraction 


X 


30 

Requirements testing plan exists 


X 


31 

Requirements testing is formal 


X 


32 

All requirements have priorities 



X 

33 

All requirements must be implemented 



X 

34 

Requirements are tested 



X 

35 

All requirements are equally important 



X 

36 

At least some requirements have priorities 


X 


37 

All requirements are traceable 


X 


38 

Traceability not important 



X 

39 

Each requirement has an author 


X 


40 

Who authored requirement is not important 



X 

41 

Initial set of requirements to be implemented, no requirements creep 



X 

42 

Structured and tracked changes to requirements only 



X 

43 

Change is inevitable, changes allowed at all times 



X 

44 

Change is inevitable, but changes limited 



X 

45 

Requirements control funding 



X 

46 

Requirements history kept 

X 



47 

Baseline established for requirements at some point prior to develop 



X 

TOTAL SCORING 

KJ 
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3. Estimation/Planning Questionnaire Responses 


No. Estimation/Planning Questionnaire - Total: Block f 

Yes 

No 

N/A 

1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

X 



2 

Measure used for various product elements (modules, components, CSCI) 

X 



3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 

X 



4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

X 



5 

Product matrics tracked and updated hroughout program execution 

X 



6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 


X 


7 

Time measure process metric used (cycle time) 

X 



8 

Process metrics tracked and updated throughout program execution 




9 

Program cost estimations made from product or process metrics 


X 


10 

Program cost extimations tracked and updated to reflect progress/changes 

X 



11 

Factor analysis performed on program 


X 


12 

Program's primary purpose, including major functions and deliverables known 




13 

Work breakdown structure developed 

X 



14 

Task estimated with realistic expectations of productivity probabilities 

X 



15 

Schedules developed based on realistic expectations 

X 



16 

Schedules tracked and updated based on new information 

X 



17 

Detailed activity lists used for clearly defined completed/not completed tasks 




18 

Quality assurance plan orsimilarto aid in detecting defects earlyin program 

X 



19 

COCOMO estimates performed 

X 



20 

CSCI clearly defined and tasked 




21 

Estimates completed ad hoc 


X 


22 

Gantt charts used and updated 

X 



23 

Resource estimations (working hrs, job categories, task activities) done 

X 



24 

Earned value established 

X 



25 

Earned value tracked throughout program 

X 



26 

Quality expectations established for product with users and stakeholders 

X 



27 

Critical path for program tasks developed and tracked 

X 



28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 


X 


29 

Estimates are updated routinely 

X 



30 

Schedules are updated routinely 

X 



31 

Estimations are made by program management (top-down) 

X 



32 

Estimateions are made by program team members (bottom-up) 

X 



33 

Automated program tracking used 

X 



34 

PM usuallythorough in tracking and reporting schedules and financials 




35 

WBS developed only as data call 



X 

36 

Earned value used to track program progress 

X 



37 

PM insists on prioritizing work reduction as schedule/funding compromised by stakeholders 



~>r 

38 

Estimations are done using both top down and bottoms up approaches 



X 

39 

All program team members involved in planning process 



X 

40 

Hardware also considered in estimaation process 



X 

41 

Program history compiled 

X 



42 

System upgrades (SCR) software changes requests estimated individually 



“x - 

43 

Management duties apart of each team member's responsibilities 



X 

44 

PM dictates schedules to program team 



X 

45 

Code reviews planned in schedule 

X 



46 

Defined tangible milestones established for program tasks 

X 



47 

Test planning done at the start of the program 

X 



48 

Estimations are completed bythose performing the tasks 



“x - 

49 

Sensitivity analysis performed for program choices 



X 

50 

Software deployment planning completed 



X 

TOTAL SCORING 

38 
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People Management Questionnaire Responses 


No. People Management Questionnaire - Total: Block g_Yes No N/A 


1 

PM is accessible in person by each team member 



X 

2 

PM is accessible via email (memo, letter) by each team member 



X 

3 

PM is accessible via phone by each team member 



X 

4 

PM not only considers a person's suitability, not also desire to be on a team 



X 

5 

PM consults with each team member regarding their career goals 



X 

6 

PM regularly holds meetings to inform team of program progress 



X 

7 

PM solicits opinions from team members before making decisions 



X 

8 

PM lets teams make decisions affecting their work 



X 

9 

PM freuently makes decisions without any consultation with members 

X 



10 

PM understands the technology/language of the program 



X 

11 

PM is able to communicate with other the technical issues in the program 



X 

12 

PM prioritized problems or conflicts within the program 



X 

13 

PM assists team members in developing/advising of career path 



X 

14 

PM empowers program members to recommend hiring new team members 



X 

15 

PM empowers program members to recommend firings of other members 



X 

16 

PM specifically assigns work to each program member 



X 

17 

PM sets communication protocol 



X 

18 

PM allows unrestricted communications 



X 

19 

PM encourages or requires training for each individual 



X 

20 

PM takes control in difficult/roblem areas 



X 

21 

PM looks ahead to new programs, new upgrades of existing program 



X 

22 

PM maintains regular communications with all stakeholders 



X 

23 

PM maintains regular communications with users 



X 

24 

PM encourages program team communication with users 



X 

25 

PM encourages program team communication with stakeholders 



X 

26 

PM facilitates horizontal communication within program 



X 

27 

PM facilitates communication during integration 



X 

28 

PM holds meetings without clear objectives 

X 



29 

PM must approve all decisions within the program 

X 



30 

PM must approve ail interactions with stakeholders 

X 



31 

PM must approve all interactions with users 

X 



32 

PM makes all presentations to stakeholders/users 

X 



33 

PM is considered "flexible" in terms of program members personal issues 



X 

34 

PM, at least occasionally, schedules/promotes outside work team activities 



X 

35 

PM is readily willing to listen to program prblems and complaints 


X 


36 

PM takes action to resolve program problems and complaints 


X 


37 

PM is generally respected by stakeholders, users, and organization 




38 

PM sometimes fails to grasp important technical issues in program 

X 



39 

PM recruits program team members from outside organization 


X 


40 

PM participates in technical reviews 

X 



41 

Program personnel have clearly defined specific tasks 

X 



42 

Although individual's tasks are specific, each exposed to the "bigger picture" 


X 


43 

PM has clearly defined his/her expectations for each individual 


X 


44 

PM delegation of duties is usually seemless in execution 


X 


45 

PM acts as facilitator to solving personnel conflicts 


X 


46 

PM attempts to motivate individuals on the program team 


X 


47 

PM clearly spearates technical from managerial roles for individuals 



X 

48 

PM directs how he/she expects the task to be accomplished 



X 

49 

PM directs what needs to be done, but does not direct how 



X 

50 

PM attempts to spotlight individuals in the program for positive exposure 



X 

TOTAL SCORING 
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5. Risk Management Questionnaire Responses 


No. Risk Management Questionnaire - Total: Block h 

Yes 

No 

N/A 

1 

Risk Management (RM) is specifically an activity in the program 

X 



2 

RM is formal and documented 

X 



3 

A specific RM Ian exists 

X 



4 

RMis required in the program, but not used during the program 


X 


5 

RMis done prior to the program execution 

X 



6 

RMis done by an outside entity to the development 


X 


7 

RMis done internally only 


X 


8 

RMis both internally performed and externally assessed 

X 



9 

RM planning occurs during or after major milestones in the program 

X 



10 

Risk Assessment is only a management function 


X 


11 

RM is informal or non existent 


X 


12 

There is a RM plan, but it is not updated or tracked 

X 



13 

Risks are only generalized 


X 


14 

Each risk is delineated 

X 

X 


15 

Each risk has a consequence 

X 

X 


16 

Each risk has a likelihood rating of some sort 

X 



17 

Each risk has a mitigation strategy 

X 



18 

Risk Management is automated 


X 


19 

Risks are tracked 

X 



20 





21 

Regret analysis performed 

X 



22 

RM drives decisions in the program 


X 


23 

Risks have probabilities 

X 



24 

Risk Management is ad hoc 


X 


25 

RM information is shared with all stakeholders (as appropriate) 

X 



26 

Risks are weighed relative to other program risks 

X 



27 

Risk Assessment is a program team activity 

X 



28 

Risk Assessment done prior to program start 

X 



29 

Risk Assessment includes personnal risk 


X 


30 

RMuses tools, but depends on human decisions 

X 



31 

Risk assessment includes cost risks 

X 



32 

Risk Assessment includes schedule risks 

X 



33 

Risk Assessment includes technology risks 


X 


34 

Risk Assessment is briefed organization structure above program manager 



X 

35 

Risk Assessment includes requirements risks 



X 

36 

Risk Assessment includes user risks (too little involvement of user) 

X 



37 

Risk Assessment includes documentation risks 


X 


38 

Risk Assessment includes integration risks 

X 



39 

Risk Assessment includes interface risks (non-standard) 

X 



40 

Risk Assessment includes continuing requirements change (feature creep) 

X 



41 

Risk Assessment includes dependent projects/programs risks 


X 


42 

Documentation proof exists to demonstrate following risk management plan 


X 


43 

High rish have measured tracking (high profile status) 

X 



44 

Organizational history used to search for risks 

X 



45 

Other organizational checklists used for risk assessment 

X 



46 

Internal organizational checklists used for risk assessment 

X 



47 

Risk Assessment information contributed to internal or other database 

X 



48 

Risk Assessment includes internal organization risks 

X 



49 

Risk Assessment includes stakeholder risks 

X 



50 

No risk management needed; program is straightforward & understood 


X 


TOTAL SCORING 

£21 

El 
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6. Pair Choices Responses 

Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

X 

informal requirement list 

w ritten requirements 

X 

oral requirements 

requirements informal, but recorded 

X 

requirements not recorded 

requirements as part of an SRS (or other formal repository) 

X 

requirements informally recorded 

requirements taken as is from customer 


look to reformulate, interview in-depth, or otherwise re-validate 

only one development strategy used 

X 

strategies not consistent, used at different times 

stakeholders as part of requirements development 

X 

stakeholders approving requirements after formulated by development team 

requirements are testable 

X 

requirements have no test plans 

informal test plan or no test plan 


formal test plan 

test team involved w ith requirements 

X 

no test team input or plans during requirements development 

only a percentage of requirements present in baseline 


baseline must contain all requirements 

requirements documentation has hierarchical structure 


all requirements must be implemented 

requirements have listed responsible party 


requirements origin not important 

requirements documentation have versions 


no requirements history 

requirements have specific attribute values 


requirements all rank evenly 

funding controls requirements definition 


requirements definition controls funding 

reqquirements are top dow n 

X 

requirements are bottom up 

users/stakeholders are identified and interviewed (market survey) 

X 

no special consideration to identify users/stakeholders 

each requirement has a singular concept 


some requirements are compound statements 

requirements definition minimized when funding short 

X 

program scope may reduce, but requirements definition completed 

requirements extraction has formal process 

X 

requirements extraction ad hoc 

change procedures formal 

X 

change procedures ad hoc 

users/stakeholders somehow involved in requirements definition 


program team only involved in requirement definition 

management sets requirements for developers 

X 

developers at least partially involved in setting requirements 

requirements changed at least once since baseline established prior to new version 

X 

requirements in baseline has not changed prior to new version or upgrade 

no ranking of requirements 

X 

requirements have priorities assigned 

use-case diagrams (or other models or scenario developments) 


no models used for requirements extraction 

requirements changes informal 


requirements changes formal 

plan to "freeze" requirements at some designated milestone 


no provision for "freezing" requirements 

requirements must be traceable 

X 

origin of requirements not important 

requirements must be testable 

X 

system developed must be testable 

test plans to determine requirements implemented 


no test plans needed for requirements verification 

requirements have priorities in implementation 


all requirements must be implemented 

some requirements have multiple statements or ideas 


one idea, one statement per requirement 


Requirements Management (page 1 of 2) score | 29 
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Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


I ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) ( 

requirements first, then initial development work 


initial development work then requirements 


requirements documentation driving development 


requirements documentation developed in parallel/after development 


user feedback considered during development 


after development starts, user feedback serves as input to new work 


change management procedures used strictly 


change management procedures as guidance only 


design decisions prior to or in parallel to requirrements development 


design decisions only after approved requirements stabilized 


requirements summarized wht we have developed 


requirements are the blueprint for development 


length of time for requirements work greater than development work 


length of time for requirements work less than development work 


requirements have design detail 


no design detail in requirements 


requirements creep to be avoided 


requirements creep o.k., but need to be controlled 


freeze requirements at some point 


requirements are fluid throughout development 


formal change procedure 


informal change procedure 


change management plan 


no change management plan 


requirements ambiguity always present to some extent 


requirements ambuiguity unacceptable at any level 


testing considered up frornt during requirements determination 


testing considered down the line during development 


requirements development team members different from implementation 


those working on requirements, work on implementation 


start implementation as early as possible to help define requirements 


requirements must be defined prior to any implementation work 


j ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWAY, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 1 

develop prototype, then determine requirements 

X 

determine requirements prior to any development work 


requirements testing done after each iteration 

X 

no testing 


individual changes as necessary 

X 

only block changes made 


development team decides on changes after iteration 


users involved with changes 

X 

changes based on feedback only from user for correction of problems 

X 

changes to upgrade system and correct problems 


funding controls changes and change procedures 

X 

changes control funding 


requirements documentation finalized prior to development 

X 

requirements fluid throughout development (only freeze at end) 


requirements test plans completed prior to development 

X 

requirements test plans completed after development 


requirements first, then initial development work 

X 

initial development work then requirements 


use development effort to learn more about requirements 

X 

define all requirements prior to coding anything 


requirements ambiguity always present to some extent 

X 

requirements ambiguity unacceptable at any level 


requirements have design detail 

X 

no design detail in requirements 


user feedback considered during development 

X 

after development starts, user feedback serves as input to new work 


get something to users as soon as possible for evaluation 

X 

make sure it is complete before releasing 


management dictates requirements 


development team visually represent requirements through rapid prototyping 

X 

new requirements allowed after initial requirements defined 

X 

new requirements not allowed 



Requirements Management (pg 2 of 2) score [16] +pg 1 score [29] = TOTAL SCORE [45] Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

X 

no estimates 

formal derivation of product metric for estimation of size 

X 

ad hoc size estimation 

ad hoc process evaluation 

X 

formal derivation of at lest one process metric 

develop work breakdown structure (WBS) 

X 

assign work as needs arise 

estimates are developed to fulfill a data call only 


use estimates to plan program 

use estimates to sell program only 


estimates are useful to the project tema for planning purposes 

resource evaluations made for program 

X 

no resource evaluation for planning 

use both bottom up & top down for estimate, use one stakeholders like 

X 

use both bottom up & top down and evaluate significant differences 

estimates made and not updated 


estimates updated throughout program 

resources estimations used to adjust product size estimate 

X 

estimations made irregardless of resources available 

estimations made to fit budget 


budget made from estimations 

estimations compromised to get program 

X 

rather risk loss of program than compromise confident estimations 

cycle time estimations 

X 

no cycle time estimations 

event count estimations 

X 

no event count estimations 

lines of code (LOC) estimation 

X 

no LOC estimation 

function pont (FP) estimation 

X 

no FP estimation 

estimates by algorithmic methods 

X 

estimates by analogy 

expert judgement for estimates 

X 

ad hoc estimates 

estimates by algorithmic methods 

X 

ad hoc estimates 

expert judgement for estimates 

X 

estimates by analogy 

ad hoc estimates 

X 

estimates by analogy 

bottom up estimates 

X 

expert judgement 

top down estimates 

X 

expert judgement 

ad hoc estimates 

X 

any other estimate process 

fuzzy logic estimating method 

X 

no formal estimation methodology 

WBS development from estimates 

X 

WBS development in parallel or prior to estimation completion 

critical path of program determined 

X 

tasks developed but no path is identified 

estimators are program team members 

X 

estimators are outside program team 

management only on estimations 

X 

all team members involved in estimation process 

estimates updated at reviews 

X 

no updates of estimates 

estimates updated at reviews 

X 

estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 

X 

estimate procedures change 

stakeholders are part of estimation process 

X 

stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

X 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

X 

important to have WBS as guide, not rigid implementation 


Estimation/Planning Management (page 1 of 2) score | 27 | 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

X 

estimates for program initiation only 

system upgrades (SCR) software change requests estimated individually 

X 

systems upgrades estimated as w hole 

estimates for on-gong resources needed to maintain s/w 

X 

estimates for maintenance not done 

informal re-estimates during development 


formal re-estimates at pre-defined milestones 

formal re-estimates w hen amendment changing the system is introduced 

X 

informal re-estimates w hen amendment changing the system 

person in-charge of estimation walks in a managers office to get an opinion 

X 

meeting(s) organized for purpose of performing cost estimations 

factor analysis prior to commencement of program 

X 

none done 

change control procedures set in place 

X 

no set procedures 

elapsed time and actual work time estimates 

X 

one or the other or neither 

no schedule created 


scheudle created 

schedule not updated 


schedule updated 

schedule follow ed 

X 

schedule not followed 

tasks identification arises as program progresses 

X 

detailed level tasks identified prior to program initiation 

scope of program understood by all 

X 

scope not explicitly defined 

quality factors and criteria identified 

X 

no explicit quality factors defined 

no project tracking tools used 

X 

project tracking tools used 

CSCIs identified and tasked 

X 

CSCIs not explicitly identified 

expectations are managed via estimations 

X 

estimations are made to fit preconceived expectations 

no cost schedule developed 


cost schedule developed 

no resource schedule developed 


resource schedule developed 

team members, management know at any time if in budget & schedule 

X 

exact budget & schedule status somew hat unclear to at least some 

individual program phases are estimated 

X 

only top level program estimated 

stakeholders/users emphasis understood-quickto field or all complete 

X 

program management sets delivery tradeoffs w ithout outside input 

testing planned with initial program planning 

X 

testing not in initial planning 

documentation not considered ininitial planning 

X 

documentation part of initial planning 

hardw are considered in estimations 

X 

software only considered 

no formal schedule/cost tracking 


formal procedures established for tracking cost and schedule 

earned value set up 

X 

earned value not used 

estimations omit documentation planning 

X 

documentation in estimates 

training omitted in estimates 

X 

training part of estimates 

earned value set up, but not tracked 

X 

earned value tracked 

detailed planning done w ith incomplete set of requirements 

X 

detailed planning done w ith detailed set of requirements 

complete infrastructure support mechanism understood for estimations 

X 

no consideration of infrastructure done for estimations 

team possibilities considered for planning of program 

X 

no consideration for outside teaming possibilities 

work breakdow n structure (WBS) set up 

X 

no WBS completed 


Estimation/Planning Management (pg 2 of 2) score [27] +pg 1 score [27] = TOTAL SCORE [54] Enter on QMM scoresheet blk b. 
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Pair choice section THREE: (People Management) choose most applicable 
Human Resources 


program team members have clearly deined, segmented roles 


formal team building procedures are used 


program manager flexible regarding work hours 


big picture conveyed to all team members by program management 


people issues dealt with primarily through indirect methods (email, memo, etc) 


training is required and planned on a regular basis 


each team member is educated on and understands overall program and their roles 


consideration for team members' career goals are reflected in assignments 


team members assignments and responsibilities are mostly dictated by PM 


management leads in problem solving 


management welcomes problems as challenges and opportunities 


team members participate in performance evaluations of peers 


management reinforcement feedback sparse and inconsistent, if any 


management provides basic needs of office facilities fairly well 


working conditions are fairly comfortable, time off policy fairly good 


Communication: 


communications primarily written (email) 


detailed instructions: oral presentation, follow-up email 


formal communication protocol 


external vertical communications restricted 


coders notebook weekly accomplishment reports required 


user-coder relationship established, encouraged, and mediated 


meetings structured to minimize waster time 


meetings have agenda, objectives, and conclude with action items 


program management and coder communication face to face 


program team updated regularly regarding organizational & program status 


open communications is encouraged 


program manager accessible for discussions 


program management (PM) is viewed as separate from team 


management regularly holds team meetings 


meetings are structured with definite goals and objectives 


program management is generally easy to reach and talk to 


team-program manager relationship adult-adult 


schedules are spontaneous and poorly communicated 


work is seen as complex processes involving team working together 


action items often is poorly disseminated and usually not followed through 


team members require frequent clarifications by PM for assigned tasks 


term of the two for each row (page 1 of 2): 


X |work responsibilities are shared 


no formal team building emphasized 


program manager maintains strict standards for work hours 


program management focuses on the partitioned tasks with team 


people issues dealt with primarily through direct methods (face-to-face) 


X |training is ad hoc 


team members only know their respective areas 


team members must adapt to tasks that are assigned 


X assignments and responsibilities are discussed and agreed upon with PM 


X |management facilitates and lets team lead in problem solving 


management views problems as obstacles and grounds for punishment 


Personnel evaluations are strictly PM responsibility 


X management provides timely reinforcement feedback for positive behaviors 


office facilities are a drawback to working in the program 


X working conditions and time off policy is inconsistent and difficult at times 


X Icommunications primarily verbal (face-to-face) 


email only 


informal communications 


external vertical communication allowed 


not required 


user-coder interaction minimized 


meetings unstructured and open ended 


meeting agenda fluid and open ended 


program management and coder communication primarily email 


X |meetings infrequently scheduled 


communication hrough chain of command only is encouraged 


program manager difficult to get an appointment to see 


X | PM mixes with team frequently 


meetings are sporadic 


meetings are informal 


PM is usually hard to get a hold of and difficult to talk to 


team-program management relationship parent-child 


schedules must be fixed and rigidly followed and formally reported 


work broken into pieces with minimal team member interaction 


X action items communicated and followed through thoroughly 


X |team members rarly require clarifications by PM for assigned tasks 


XX XX XX XX XX X XX 




















































































































Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

X 

short tern program and immediate w ork focus 


lead through personal attention to others 


action-oriented leadership approach 

X 

run as much of the organization as possible 

X 

let team make decisions as much as possible 


direct and domineering style 

X 

encourage independence in others 


traditional leaders respect hierarchy 

X 

do w hat needs to be done 


w in cooperation rather than demand it 

X 

tough-minded with others 


act strongly and forcefully in the field of ideas 

X 

prefer to lead other independent types while seeking auto no my for self 


consults with team members to find solutions to problems 

X 

consults team members to get validation of PMs predetermined solutions 


keep people w ell informed 

X 

only as much know ledge as necessary for their w ork 


make things happen by focusing on the immediate problems 


long range focus and de-emphasize current problems 

X 

manage others loosely and prefer minimal supervision 

X 

follow traditional procedures and rules conscientiously 


leadership, management decisions exclusively by program management 

X 

program management makes decisions but gets inputs from team 


team-program manager relationship adult-adult 

X 

team-program management relationship parent-child 


program management makes decisions but gets inputs from team 

X 

all program team members responsible for program decisions 


w hen a problem arises: management takes over to solve it 

X 

management lets the team solve the problems 


leadership is do as 1 say, not do as 1 do 

X 

leadership by example 


program expectation not influenced by PM 

X 

program expectation managed by FM 


PM gives freedom to team, but has no mentoring for members (abdication) 

X 

FM empowers teams by mentoring members to be leaders 


promgram management waits and sees what happens then plans 


management plans far in advance 

X 

program management is constantly reacting to emergencies 

X 

management is one step ahead of problems 


facilitative approach to solving problems 

X 

take charge readily and often 


program management is complex, takes much time to understand 


management is simple, easy to figure out 

X 

program management prefers to plunge right in 

X 

takes time to separate things to be done and order of doing them 


program management reacts spur of the moment 

X 

methodically follows plans 



Technical Competency of the Program Manager: 


FM has technical experience particular to the particular s/w program 


FM relies on team members solely 

X 

FM participates in technical review s 

X 

FMonly in non-technical reviews 


FM participates in making technical decisions w hen problems arise 

X 

FM delegates technical questions 


FMdoes not get involved discussing technical options 

X 

FM contributes to technical options being discussed 


FMdoes not review technical options and decisions 


FM reviews technical options and decisions 

X 

PM actively attempts to keep up-to-date with current techno logy and standards 


FM is removed from cutting edge technology issues 

X 

PM receives technical periodicals and occasionally references applicable articles 


FM doesn't read periodicals nor reference current articles to team 

X 

FM doesn't have technical background (or education) 


FM has technical background (or education) 

X 

team members avoid FM w hen they need technical advice 

X 

team members generally consider talking to FM regarding technical issues 



HR [4] + Comm. [4] + Leadership [4] + Tech. Competency [4] = People Mgmt. score [16] Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

X 

RM is informal, if at all 

a risk management plan exists 

X 

no risk management plan is developed 

RM is more of a data call than a useful document 

X 

RM drives decisions on the program 

RM is done prior to the program beginning 

X 

RM is done prior and during program execution 

RM is only done during the program execution 

X 

RM is done prior and during program execution 

risks are generalized through the whole program 


risks are categorized 

risk management is done internally, only 


an outside organization also contributes to the RM process 

risk is a management function 


risk is a program team function 

risks are precisely articulated 


risks are generalized, if at all 

each risk has a consequence 

X 

consequences are generalized, if at all 

a mitigation strategy is completed for each risk 

X 

mitigation strategy is generalized, if at all 

contingency plans are developed for a RM plan 

X 

contingency plans are ad hoc as problems arise in the program 

risks are anticipated 

X 

if problems arise, management will deal with it 

the program doesn't have any risk 


programs that do not have risk, have problems 

risk management is automated 


risk management may use tools, but depend on human input 

risks are assigned probabilities 

X 

probabilities are not relevant for RM 

all risks are potential problems, relative priorities for risks are not useful 


risks are weighed relative to other program risks and thus prioritized 

risk management information is only shared internally 

X 

risk management information is shared with all stakeholders 

risk analysis uses ordinal rankings 


risk analysis uses actual measurements with a mathematical model 

regret analysis used 

X 

no regret analysis done 

attach probabilities to future events 

X 

no probabilities associated with future events 

assessing risks with mechanical meethods 


risks should be compared to other risks and sorted 

risk status tracked 

X 

not tracked 

technical risks examined 

X 

no technical risks examined 

process risks examined 

X 

no process risks examined 

product risks examined 

X 

no product risks examined 

stakeholder/user risks examined 

X 

no examination of stakeholder/user risks 

checklists used to identify risks 

X 

no checklists used 

risks are tracked 

X 

no tracking or monitoring of risks 

each risk has an impact 

X 

no impact analysis of risk 

each risk has a mitigation plan 


no individual risk mitigation 

risks monitored by priority 

X 

no special attention to track higher priority risks 

risk assessment is formalized 

X 

no formal risk assessment 

risk control is formalized 

X 

no formal risk control 

integration risks not considered 

X 

integration risks examined 


Risk Management (page 1 of 2) score | 28 | 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

X 

no cost risks examined 


unforeseen risks have occurred in program 

X 

any risk that came up had been identified previously 


personnel risks examined 


no personnel risks examined 

X 

estimation risks examined 

X 

no estimation risks examined 


planning risks examined 

X 

no planning risks examined 


requirements risks examined 

X 

no requirements risks examined 


resource risks examined 

X 

no resource risks examined 


risk management plan updated regularly 

X 

no regular risk management plan updates 


risks charted 

X 

risks not charted 


performance risks examined 

X 

performance risks not examined 


program management self risks examined 

X 

no program management risks examined 


risk from program constraints examined 

X 

no program constraint risks examined 


each category of risks are prioritized 

X 

no prioritization 


each category of risks are evaluated for impact 

X 

no impact analysis performed 


each category of risks have control strategy 

X 

no control strategy 


documentation risks examined 

X 

no documentation risks examined 


regret matrix tracked 

X 

no regret matrix or not tracked 


communication of risk activities are facilitated 

X 

no facilitation or promotion of communication of risk activities 


taxonomy-based questionnaire used to identify risks 

X 

taxonomy-based questionnaire not used 


associated hardware risks examined 


no consideration for hardware risks 

X 

integration risks examined 

X 

integration risks not examined 


communication risks examined 

X 

communication risks not examined 


leadership risks examined 

X 

leadership risks not considered 


risk avoidance considered for certain risks 

X 

risk avoidance not considered for risks 


risk documentation forms used 

X 

no risk documentation forms used 


dependency risks examined 

X 

no dependency risks examined 


alternatives like risk avoidance considered for high risk items 


no consideration of risk avoidance 

X 

documented risk statements use a condition-consequence type format 

X 

condition-consequence of risk statements not clearly defined 


no assignment of ownership of risk mitigation action 


each risk mitigation action is assigned to an individual for resolution 

X 

calculation of risk exposure made (probability X loss, for each risk) 

X 

no risk exposure calculations 


oral communication of risks only 

X 

risks written in a way that communicates nature and status of factors 

X 

triggers used to quantify risk conditions present 


risk conditions present are all subjective 

X 

risk "czar" in program for monitoring risks 

X 

no special positions/responsibilities for risk monitoring 


post-program review completed (scheduled) for unanticipated problems ID 

X 

no post-program reviews completed or scheduled 


no schedule risks examined 


risks to schedule investigated 

X 


Risk Management (pg 2 of 2) score [28] +pg 1 score [33] = TOTAL SCORE [61] Enter on QMM scoresheet blk d. 
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G. 


SCORING 


1. Requirements Management Questionnaire 


No. 

Requirements Management Questionnaire 

Yes 

No 

N/A 

1 

PM chose to have a formal requirements list 

1 

0 

0 

2 

Requirements recorded in some way 

2 

-1 

0 

3 

Written requirements were part of some formal document 

1 

0 

0 

4 

Written requirements were informal 

1 

2 

0 

5 

At least some requirements were oral only 

-2 

1 

0 

6 

All stakeholders were identified 

2 

-1 

0 

7 

All stakeholders participated in the requirements extraction 

2 

0 

0 

8 

Some stakeholders participated in the requirements extraction 

1 

0 

0 

9 

Management extracted requirements, no stakeholder involvement 

1 

2 

1 

10 

Management passed requirements to development team 

1 

0 

0 

11 

Stakeholders not involvved in Management extraction, but approved 

-1 

0 

0 

12 

Management gets inputs from stakeholders, then develops requirements 

1 

0 

1 

13 

Developers work informally with users to arrive at requirements 

1 

0 

0 

14 

Same as 13, but management oversees and formalizes 

2 

0 

0 

| If a waterfall or sequential development strategy: 1 

15 

All requirements complete before design 

1 

-3 

0 

16 

Some requirements left incomplete prior to design 

-1 

0 

0 

17 

Requirements informal prior to design effort 

-1 

0 

0 

18 

Requirements serve as input 

1 

-1 

0 

19 

Length of time for requirements work greater than development work 

2 

-1 

0 

20 

Requirements developed in parallel to design 

-1 

1 

0 

OR 

If a prototype , throwaway, or other development strategy: 




15 

Learn about requirements through development efforts 

1 

-1 

0 

16 

No coding until all requirements are defined 

-3 

1 

0 

17 

Requirements formal prior to design effort 

-1 

0 

0 

18 

Requirements serve as output 

1 

-1 

0 

19 

Requirements definition work in parallel to development efforts 

2 

-1 

0 

20 

Requirements developed in parallel to design 

1 

-1 

0 

21 

Are requirements frozen at some phase 

1 

-1 

0 

22 

Change management exists 

3 

-3 

0 

23 

Change management is formal 

1 

0 

0 

24 

Project strategy is consistent throughout development 

1 

0 

0 

25 

Requirements are updated 

1 

0 

0 

26 

Configuration Management (CM) exists 

3 

-3 

0 

27 

CM is formal 

1 

0 

0 

28 

Requirements are testable 

2 

-2 

0 

29 

Requirements testing considered/implemented during extraction 

2 

0 

0 

30 

Requirements testing plan exists 

2 

0 

0 

31 

Requirements testing is formal 

1 

0 

0 

32 

All requirements have priorities 

2 

-2 

0 

33 

All requirements must be implemented 

0 

1 

0 

34 

Requirements are tested 

1 

-1 

0 

35 

All requirements are equally important 

0 

1 

0 

36 

At least some requirements have priorities 

1 

0 

0 

37 

All requirements are traceable 

1 

0 

0 

38 

Traceability not important 

0 

1 

0 

39 

Each requirement has an author 

1 

0 

0 

40 

Who authored requirement is not important 

0 

1 

0 

41 

Initial set of requirements to be implemented, no requirements creep 

0 

1 

0 

42 

Structured and tracked changes to requirements only 

1 

-1 

0 

43 

Change is inevitable, changes allowed at all times 

-1 

1 

0 

44 

Change is inevitable, but changes limited 

1 

0 

0 

45 

Requirements control funding 

1 

0 

0 

46 

Requirements history kept 

1 

-1 

0 

47 

Baseline established for requirements at some point prior to develop 

2 

-2 

0 


TOTAL SCORING 
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2, 


Estimation/Planning Questionnaire 


No. Estimation/Planning Questionnaire_Yes No N/A 


1 

A volume product metric used (LOC, # of files, # of screens, pages of doc) 

1 

0 

0 

2 

Measure used for various product elements (modules, components, CSCI) 

1 

0 

0 

3 

Product measures made by phase (amt at implementation, LOC changed at unit test) 

1 

0 

0 

4 

Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) 

1 

0 

0 

5 

Product matrics tracked and updated hroughout program execution 

2 

-1 

0 

6 

Event count process metric used (# defects in test, reqmt changes, milestones met) 

1 

0 

0 

7 

Time measure process metric used (cycle time) 

1 

0 

0 

8 

Process metrics tracked and updated throughout program execution 

2 

-1 

0 

9 

Program cost estimations made from product or process metrics 

1 

0 

0 

10 

Program cost extimations tracked and updated to reflect progress/changes 

1 

0 

0 

11 

Factor analysis performed on program 

1 

0 

0 

12 

Program's primary purpose, including major functions and deliverables known 

2 

-1 

0 

13 

Work breakdown structure developed 

2 

-1 

0 

14 

Task estimated with realistic expectations of productivity probabilities 

1 

-1 

0 

15 

Schedules developed based on realistic expectations 

1 

-1 

0 

16 

Schedules tracked and updated based on new information 

1 

-1 

0 

17 

Detailed activity lists used for clearly defined completed/not completed tasks 

1 

-1 

0 

18 

Quality assurance plan or similar to aid in detecting defects early in program 

1 

-1 

0 

19 

COCOMO estimates performed 

1 

-1 

0 

20 

CSCI clearly defined and tasked 

2 

-1 

0 

21 

Estimates completed ad hoc 

-2 

0 

0 

22 

Gantt charts used and updated 

1 

-1 

0 

23 

Resource estimations (working hrs, job categories, task activities) done 

1 

-1 

0 

24 

Earned value established 

2 

-1 

0 

25 

Earned value tracked throughout program 

2 

0 

0 

26 

Quality expectations established for product with users and stakeholders 

1 

-1 

0 

27 

Critical path for program tasks developed and tracked 

2 

-1 

0 

28 

Measure of effectiveness (MOE) or Figure of merit established and tracked 

1 

0 

0 

29 

Estimates are updated routinely 

2 

-1 

0 

30 

Schedules are updated routinely 

2 

-1 

0 

31 

Estimations are made by program management (top-down) 

1 

0 

0 

32 

Estimateions are made by program team members (bottom-up) 

2 

0 

0 

33 

Automated program tracking used 

1 

0 

0 

34 

PM usually thorough in tracking and reporting schedules and financials 

1 

-1 

0 

35 

WBS developed only as data call 

-1 

0 

0 

36 

Earned value used to track program progress 

2 

-1 

0 

37 

PM insists on prioritizing work reduction as schedule/funding compromised by 
stakeholders 

1 

-1 

0 

38 

Estimations are done using both top down and bottoms up approaches 

2 

-1 

0 

39 

All program team members involved in planning process 

2 

-1 

0 

40 

Plardware also considered in estimaation process 

1 

-1 

0 

41 

Program history compiled 

1 

0 

0 

42 

System upgrades (SCR) software changes requests estimated individually 

1 

-1 

0 

43 

Management duties apart of each team member's responsibilities 

-1 

1 

0 

44 

PM dictates schedules to program team 

-1 

0 

0 

45 

Code reviews planned in schedule 

1 

-1 

0 

46 

Defined tangible milestones established for program tasks 

2 

-1 

0 

47 

Test planning done at the start of the program 

1 

-1 

0 

48 

Estimations are completed by those performing the tasks 

1 

-1 

0 

49 

Sensitivity analysis performed for program choices 

1 

-1 

0 

50 

Software deployment planning completed 

1 

-1 

0 


TOTAL SCORING 
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3 


People Management Questionnaire 


No. People Management Questionnaire_Yes No N/A 


1 

PM is accessible in person by each team member 

1 

0 

0 

2 

PM is accessible via email (memo, letter) by each team member 

1 

0 

0 

3 

PM is accessible via phone by each team member 

1 

0 

0 

4 

PM not only considers a person's suitability, not also desire to be on a team 

1 

0 

0 

5 

PM consults with each team member regarding their career goals 

1 

0 

0 

6 

PM regularly holds meetings to inform team of program progress 

2 

-1 

0 

7 

PM solicits opinions from team members before making decisions 

2 

-1 

0 

8 

PM lets teams make decisions affecting their work 

1 

0 

0 

9 

PM freuently makes decisions without any consultation with members 

-2 

2 

0 

10 

PM understands the technology/language of the program 

1 

0 

0 

11 

PM is able to communicate with other the technical issues in the program 

1 

-1 

0 

12 

PM prioritized problems or conflicts within the program 

1 

0 

0 

13 

PM assists team members in developing/advising of career path 

1 

-1 

0 

14 

PM empowers program members to recommend hiring new team members 

1 

-1 

0 

15 

PM empowers program members to recommend firings of other members 

1 

-1 

0 

16 

PM specifically assigns work to each program member 

1 

-1 

0 

17 

PM sets communication protocol 

1 

0 

0 

18 

PM allows unrestricted communications 

1 

0 

0 

19 

PM encourages or requires training for each individual 

1 

-1 

0 

20 

PM takes control in difficult/roblem areas 

1 

0 

0 

21 

PM looks ahead to new programs, new upgrades of existing program 

1 

0 

0 

22 

PM maintains regular communications with all stakeholders 

2 

-1 

0 

23 

PM maintains regular communications with users 

2 

-1 

0 

24 

PM encourages program team communication with users 

1 

-1 

0 

25 

PM encourages program team communication with stakeholders 

1 

-1 

0 

26 

PM facilitates horizontal communication within program 

1 

-1 

0 

27 

PM facilitates communication during integration 

1 

-1 

0 

28 

PM holds meetings without clear objectives 

-1 

2 

0 

29 

PM must approve all decisions within the program 

-1 

1 

0 

30 

PM must approve ail interactions with stakeholders 

-1 

1 

0 

31 

PM must approve all interactions with users 

-1 

1 

0 

32 

PM makes all presentations to stakeholders/users 

0 

1 

0 

33 

PM is considered "flexible" in terms of program members personal issues 

1 

0 

0 

34 

PM, at least occasionally, schedules/promotes outside work team activities 

1 

0 

0 

35 

PM is readily willing to listen to program prblems and complaints 

1 

-1 

0 

36 

PM takes action to resolve program problems and complaints 

1 

-1 

0 

37 

PM is generally respected by stakeholders, users, and organization 

1 

-1 

0 

38 

PM sometimes fails to grasp important technical issues in program 

-1 

1 

0 

39 

PM recruits program team members from outside organization 

1 

-1 

0 

40 

PM participates in technical reviews 

-1 

1 

0 

41 

Program personnel have clearly defined specific tasks 

0 

1 

0 

42 

Although individual's tasks are specific, each exposed to the "bigger picture" 

2 

-1 

0 

43 

PM has clearly defined his/her expectations for each individual 

2 

-1 

0 

44 

PM delegation of duties is usually seemless in execution 

1 

0 

0 

45 

PM acts as facilitator to solving personnel conflicts 

2 

-1 

0 

46 

PM attempts to motivate individuals on the program team 

2 

-1 

0 

47 

PM clearly spearates technical from managerial roles for individuals 

0 

1 

0 

48 

PM directs how he/she expects the task to be accomplished 

0 

1 

0 

49 

PM directs what needs to be done, but does not direct how 

2 

-1 

0 

50 

PM attempts to spotlight individuals in the program for positive exposure 

2 

-1 

0 

TOTAL SCORING 
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4, 


Risk Management Questionnaire 


No. Risk Management Questionnaire_Yes No N/A 


1 

Risk Management (RM) is specifically an activity in the program 

4 

-4 

0 

2 

RM is formal and documented 

3 

-3 

0 

3 

A specific RM Ian exists 

2 

-2 

0 

4 

RM is required in the program, but not used during the program 

-1 

1 

0 

5 

RM is done prior to the program execution 

1 

0 

0 

6 

RM is done by an outside entity to the development 

1 

0 

0 

7 

RM is done internally only 

0 

1 

0 

8 

RM is both internally performed and externally assessed 

1 

-1 

0 

9 

RM planning occurs during or after major milestones in the program 

1 

-1 

0 

10 

Risk Assessment is only a management function 

0 

1 

0 

11 

RM is informal or non existent 

-1 

1 

0 

12 

There is a RM plan, but it is not updated or tracked 

1 

0 

0 

13 

Risks are only generalized 

-1 

0 

0 

14 

Each risk is delineated 

1 

0 

0 

15 

Each risk has a consequence 

1 

0 

0 

16 

Each risk has a likelihood rating of some sort 

1 

0 

0 

17 

Each risk has a mitigation strategy 

1 

0 

0 

18 

Risk Management is automated 

1 

0 

0 

19 

Risks are tracked 

2 

-2 

0 

20 





21 

Regret analysis performed 

2 

0 

0 

22 

RM drives decisions in the program 

3 

-2 

0 

23 

Risks have probabilities 

1 

0 

0 

24 

Risk Management is ad hoc 

-3 

0 

0 

25 

RM information is shared with all stakeholders (as appropriate) 

1 

0 

0 

26 

Risks are weighed relative to other program risks 

1 

0 

0 

27 

Risk Assessment is a program team activity 

1 

0 

0 

28 

Risk Assessment done prior to program start 

2 

-1 

0 

29 

Risk Assessment includes personnal risk 

1 

-1 

0 

30 

RM uses tools, but depends on human decisions 

2 

-1 

0 

31 

Risk assessment includes cost risks 

1 

0 

0 

32 

Risk Assessment includes schedule risks 

1 

0 

0 

33 

Risk Assessment includes technology risks 

1 

-1 

0 

34 

Risk Assessment is briefed organization structure above program manager 

1 

-1 

0 

35 

Risk Assessment includes requirements risks 

1 

-1 

0 

36 

Risk Assessment includes user risks (too little involvement of user) 

1 

0 

0 

37 

Risk Assessment includes documentation risks 

1 

0 

0 

38 

Risk Assessment includes integration risks 

1 

-1 

0 

39 

Risk Assessment includes interface risks (non-standard) 

1 

-1 

0 

40 

Risk Assessment includes continuing requirements change (feature creep) 

1 

-1 

0 

41 

Risk Assessment includes dependent projects/programs risks 

1 

0 

0 

42 

Documentation proof exists to demonstrate following risk management plan 

1 

0 

0 

43 

High rish have measured tracking (high profile status) 

1 

0 

0 

44 

Organizational history used to search for risks 

1 

0 

0 

45 

Other organizational checklists used for risk assessment 

1 

0 

0 

46 

Internal organizational checklists used for risk assessment 

1 

0 

0 

47 

Risk Assessment information contributed to internal or other database 

1 

0 

0 

48 

Risk Assessment includes internal organization risks 

1 

0 

0 

49 

Risk Assessment includes stakeholder risks 

2 

-1 

0 

50 

No risk management needed; program is straightforward & understood 

-3 

3 

0 

TOTAL SCORING 
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Pair Choice 


Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 1 of 2): 


formal requirement list 

2 

informal requirement list 

1 

written requirements 

2 

oral requirements 

0 

requirements informal, but recorded 

1 

requirements not recorded 

0 

requirements as part of an SRS (or other formal repository) 

2 

requirements informally recorded 

1 

requirements taken as is from customer 

0 

look to reformulate, interview in-depth, or otherwise re-validate 

2 

only one development strategy used 

1 

strategies not consistent, used at different times 

0 

stakeholders as part of requirements development 

2 

stakeholders approving requirements after formulated by development team 

1 

requirements are testable 

2 

requirements have no test plans 

0 

informal test plan or no test plan 

0 

formal test plan 

2 

test team involved with requirements 

1 

no test team input or plans during requirements development 

0 

only a percentage of requirements present in baseline 

0 

baseline must contain all requirements 

2 

requirements documentation has hierarchical structure 

1 

all requirements must be implemented 

0 

requirements have listed responsible party 

1 

requirements origin not important 

0 

requirements documentation have versions 

2 

no requirements history 

0 

requirements have specific attribute values 

1 

requirements all rank evenly 

0 

funding controls requirements definition 

0 

requirements definition controls funding 

1 

reqquirements are top down 

1 

requirements are bottom up 

2 

users/stakeholders are identified and interviewed (market survey) 

1 

no special consideration to identify users/stakeholders 

0 

each requirement has a singular concept 

3 

some requirements are compound statements 

0 

requirements definition minimized when funding short 

0 

program scope may reduce, but requirements definition completed 

1 

requirements extraction has formal process 

1 

requirements extraction ad hoc 

0 

change procedures formal 

1 

change procedures ad hoc 

0 

users/stakeholders somehow involved in requirements definition 

1 

program team only involved in requirement definition 

0 

management sets requirements for developers 

0 

developers at least partially involved in setting requirements 

1 

requirements changed at least once since baseline established prior to new version 

0 

requirements in baseline has not changed prior to new version or upgrade 

1 

no ranking of requirements 

0 

requirements have priorities assigned 

1 

use-case diagrams (or other models or scenario developments) 

2 

no models used for requirements extraction 

0 

requirements changes informal 

0 

requirements changes formal 

1 

plan to "freeze" requirements at some designated milestone 

1 

no provision for "freezing" requirements 

0 

requirements must be traceable 

1 

origin of requirements not important 

0 

requirements must be testable 

3 

system developed must be testable 

1 

test plans to determine requirements implemented 

2 

no test plans needed for requirements verification 

0 

requirements have priorities in implementation 

1 

all requirements must be implemented 

0 

some requirements have multiple statements or ideas 

0 

one idea, one statement per requirement 

2 


Requirements Management (page 1 of 2) score 


117 

































































































Pair choice section ONE: (Requirements Management) choose most applicable term of the two for each row (page 2 of 2): 


ANSWER THIS BLOCK OF QUESTIONS ONLY IF A SEQUENTIAL OR WATERFALL APPROACH IS USED FOR DEVELOPMENT (Requirements page 2 of 2) 


requirements first, then initial development work 

1 

initial development work then requirements 

requirements documentation driving development 

1 

requirements documentation developed in parallel/after development 

user feedback considered during development 

1 

after development starts, user feedback serves as input to new work 

change management procedures used strictly 

1 

change management procedures as guidance only 

design decisions prior to or in parallel to requirrements development 

0 

design decisions only after approved requirements stabilized 

requirements summarized wht we have developed 

0 

requirements are the blueprint for development 

length of time for requirements work greater than development work 

2 

length of time for requirements work less than development work 

requirements have design detail 

0 

no design detail in requirements 

requirements creep to be avoided 

1 

requirements creep o.k., but need to be controlled 

freeze requirements at some point 

1 

requirements are fluid throughout development 

formal change procedure 

1 

informal change procedure 

change management plan 

2 

no change management plan 

requirements ambiguity always present to some extent 

0 

requirements ambuiguity unacceptable at any level 

testing considered up frornt during requirements determination 

2 

testing considered down the line during development 

requirements development team members different from implementation 

0 

those working on requirements, work on implementation 

start implementation as early as possible to help define requirements 

0 

requirements must be defined prior to any implementation work 

ANSWER THIS BLOCK OF QUESTIONS ONLY IF A PROTOTYPING, THROWAWAY, SYNCHRONIZE & STABILIZE, OR OTHER STRATEGY USED 

develop prototype, then determine requirements 

1 

determine requirements prior to any development work 

requirements testing done after each iteration 

1 

no testing 

individual changes as necessary 

1 

only block changes made 

development team decides on changes after iteration 

0 

users involved with changes 

changes based on feedback only from user for correction of problems 

1 

changes to upgrade system and correct problems 

funding controls changes and change procedures 

1 

changes control funding 

requirements documentation finalized prior to development 

0 

requirements fluid throughout development (only freeze at end) 

requirements test plans completed prior to development 

1 

requirements test plans completed after development 

requirements first, then initial development work 

0 

initial development work then requirements 

use development effort to learn more about requirements 

2 

define all requirements prior to coding anything 

requirements ambiguity always present to some extent 

1 

requirements ambiguity unacceptable at any level 

requirements have design detail 

1 

no design detail in requirements 

user feedback considered during development 

1 

after development starts, user feedback serves as input to new work 

get something to users as soon as possible for evaluation 

2 

make sure it is complete before releasing 

management dictates requirements 

0 

development team visually represent requirements through rapid prototyping 

new requirements allowed after initial requirements defined 

1 

new requirements not allowed 


Requirements Management (pg 2 of 2) score 



+pg 1 score 



= TOTAL SCORE 



Enter on QMM scoresheet blk a. 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 1 of 2): 


at least one estimation method used in program 

1 

no estimates 

formal derivation of product metric for estimation of size 

1 

ad hoc size estimation 

ad hoc process evaluation 

0 

formal derivation of at lest one process metric 

develop work breakdown structure (WBS) 

1 

assign work as needs arise 

estimates are developed to fulfill a data call only 

0 

use estimates to plan program 

use estimates to sell program only 

0 

estimates are useful to the project tema for planning purposes 

resource evaluations made for program 

1 

no resource evaluation for planning 

use both bottom up & top down for estimate, use one stakeholders like 

0 

use both bottom up & top down and evaluate significant differences 

estimates made and not updated 

0 

estimates updated throughout program 

resources estimations used to adjust product size estimate 

1 

estimations made irregardless of resources available 

estimations made to fit budget 

0 

budget made from estimations 

estimations compromised to get program 

0 

rather risk loss of program than compromise confident estimations 

cycle time estimations 

1 

no cycle time estimations 

event count estimations 

1 

no event count estimations 

lines of code (LOC) estimation 

1 

no LOC estimation 

function pont (FP) estimation 

1 

no FP estimation 

estimates by algorithmic methods 

1 

estimates by analogy 

expert judgement for estimates 

1 

ad hoc estimates 

estimates by algorithmic methods 

1 

ad hoc estimates 

expert judgement for estimates 

0 

estimates by analogy 

ad hoc estimates 

0 

estimates by analogy 

bottom up estimates 

1 

expert judgement 

top down estimates 

1 

expert judgement 

ad hoc estimates 

0 

any other estimate process 

fuzzy logic estimating method 

1 

no formal estimation methodology 

WBS development from estimates 

1 

WBS development in parallel or prior to estimation completion 

critical path of program determined 

1 

tasks developed but no path is identified 

estimators are program team members 

1 

estimators are outside program team 

management only on estimations 

0 

all team members involved in estimation process 

estimates updated at reviews 

1 

no updates of estimates 

estimates updated at reviews 

0 

estimates constantly updates (in between reviews, to) 

estimate procedures stay the same 

1 

estimate procedures change 

stakeholders are part of estimation process 

1 

stakeholders brief estimations after completion 

estimates are used beyond initial selling of program 

1 

estimates are one time events, used for a specific purpose once 

WBS has objective measure of completeness 

1 

important to have WBS as guide, not rigid implementation 


Estimation/Planning Management (page 1 of 2) score 
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Pair choice section TWO: (Estimation/Planning Management) choose most applicable term of the two for each row (page 2 of 2): 


life cycle estimates 

1 

estimates for program initiation only 

0 

system upgrades (SCR) software change requests estimated individually 

1 

systems upgrades estimated as whole 

0 

estimates for on-gong resources needed to maintain s/w 

1 

estimates for maintenance not done 

0 

informal re-estimates during development 

0 

formal re-estimates at pre-defined milestones 

1 

formal re-estimates when amendment changing the system is introduced 

1 

informal re-estimates when amendment changing the system 

0 

person in-charge of estimation walks in a managers office to get an opinion 

0 

meeting(s) organized for purpose of performing cost estimations 

1 

factor analysis prior to commencement of program 

1 

none done 

0 

change control procedures set in place 

1 

no set procedures 

0 

elapsed time and actual work time estimates 

1 

one or the other or neither 

0 

no schedule created 

0 

scheudle created 

1 

schedule not updated 

0 

schedule updated 

1 

schedule followed 

1 

schedule not followed 

0 

tasks identification arises as program progresses 

0 

detailed level tasks identified prior to program initiation 

1 

scope of program understood by all 

1 

scope not explicitly defined 

0 

quality factors and criteria identified 

1 

no explicit quality factors defined 

0 

no project tracking tools used 

0 

project tracking tools used 

1 

CSCIs identified and tasked 

1 

CSCIs not explicitly identified 

0 

expectations are managed via estimations 

1 

estimations are made to fit preconceived expectations 

0 

no cost schedule developed 

0 

cost schedule developed 

1 

no resource schedule developed 

0 

resource schedule developed 

1 

team members, management know at any time if in budget & schedule 

1 

exact budget & schedule status somewhat unclear to at least some 

0 

individual program phases are estimated 

1 

only top level program estimated 

0 

stakeholders/users emphasis understood-quick to field or all complete 

1 

program management sets delivery tradeoffs without outside input 

0 

testing planned with initial program planning 

1 

testing not in initial planning 

0 

documentation not considered ininitial planning 

0 

documentation part of initial planning 

1 

hardware considered in estimations 

1 

software only considered 

0 

no formal schedule/cost tracking 

0 

formal procedures established for tracking cost and schedule 

1 

earned value set up 

1 

earned value not used 

0 

estimations omit documentation planning 

0 

documentation in estimates 

1 

training omitted in estimates 

0 

training part of estimates 

1 

earned value set up, but not tracked 

0 

earned value tracked 

1 

detailed planning done with incomplete set of requirements 

0 

detailed planning done with detailed set of requirements 

1 

complete infrastructure support mechanism understood for estimations 

1 

no consideration of infrastructure done for estimations 

0 

team possibilities considered for planning of program 

1 

no consideration for outside teaming possibilities 

0 

work breakdown structure (WBS) set up 

1 

no WBS completed 

0 


Estimation/Planning Management (pg 2 of 2) score 



+pg 1 score 



= TOTAL SCORE 



Enter on QMM scoresheet blk b. 
















































































































Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 1 of 2): 
Human Resources 


program team members have clearly deined, segmented roles 


work responsibilities are shared 


formal team building procedures are used 


no formal team building emphasized 


program manager flexible regarding work hours 


program manager maintains strict standards for work hours 


big picture conveyed to all team members by program management 


program management focuses on the partitioned tasks with team 


people issues dealt with primarily through indirect methods (email, memo, etc) 


people issues dealt with primarily through direct methods (face-to-face) 


training is required and planned on a regular basis 


1 


training is ad hoc 


each team member is educated on and understands overall program and their rolq 1 


team members only know their respective areas 


consideration for team members' career goals are reflected in assignments 


1 


team members must adapt to tasks that are assigned 


team members assignments and responsibilities are mostly dictated by PM 


assignments and responsibilities are discussed and agreed upon with PM 


management leads in problem solving 


management facilitates and lets team lead in problem solving 


management welcomes problems as challenges and opportunities 


management views problems as obstacles and grounds for punishment 


team members participate in performance evaluations of peers 


Personnel evaluations are strictly PM responsibility 


management reinforcement feedback sparse and inconsistent, if any 


management provides timely reinforcement feedback for positive behaviors 


management provides basic needs of office facilities fairly well 


office facilities are a drawback to working in the program 


working conditions are fairly comfortable, time off policy fairly good 


working conditions and time off policy is inconsistent and difficult at times 


Communication: 


communications primarily written (email) 


detailed instructions: oral presentation, follow-up email 


communications primarily verbal (face-to-face) 


email only 


formal communication protocol 


informal communications 


external vertical communications restricted 


external vertical communication allowed 


coders notebook weekly accomplishment reports required 


not required 


user-coder relationship established, encouraged, and mediated 


user-coder interaction minimized 


meetings structured to minimize waster time 


meetings unstructured and open ended 


meetings have agenda, objectives, and conclude with action items 


meeting agenda fluid and open ended 


program management and coder communication face to face 


program management and coder communication primarily email 


program team updated regularly regarding organizational & program status 


meetings infrequently scheduled 


open communications is encouraged 


communication hrough chain of command only is encouraged 


program manager accessible for discussions 


program manager difficult to get an appointment to see 


program management (PM) is viewed as separate from team 


PM mixes with team frequently 


management regularly holds team meetings 


meetings are sporadic 


meetings are structured with definite goals and objectives 


meetings are informal 


program management is generally easy to reach and talk to 


PM is usually hard to get a hold of and difficult to talk to 


team-program manager relationship adult-adult 


team-program management relationship parent-child 


schedules are spontaneous and poorly communicated 


schedules must be fixed and rigidly followed and formally reported 


work is seen as complex processes involving team working together 


work broken into pieces with minimal team member interaction 


action items often is poorly disseminated and usually not followed through 


action items communicated and followed through thoroughly 


team members require frequent clarifications by PM for assigned tasks 


team members rarly require clarifications by PM for assigned tasks 
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Pair choice section THREE: (People Management) choose most applicable term of the two for each row (page 2 of 2): 
Leadership: 


long range organizational vision 

1 

short tern program and immediate work focus 

0 

1 

1 

1 

1 

0 

1 

0 

0 

1 

0 

1 

0 

1 

1 

1 

1 

1 

1 

1 

0 

1 

1 

1 

lead through personal attention to others 

1 

action-oriented leadership approach 

run as much of the organization as possible 

0 

let team make decisions as much as possible 

direct and domineering style 

0 

encourage independence in others 

traditional leaders respect hierarchy 

0 

do what needs to be done 

win cooperation rather than demand it 

1 

tough-minded with others 

act strongly and forcefully in the field of ideas 

0 

prefer to lead other independent types while seeking autonomy for self 

consults with team members to find solutions to problems 

1 

consults team members to get validation of PM's predetermined solutions 

keep people well informed 

1 

only as much knowledge as necessary for their work 

make things happen by focusing on the immediate problems 

1 

long range focus and de-emphasize current problems 

manage others loosely and prefer minimal supervision 

1 

follow traditional procedures and rules conscientiously 

leadership, management decisions exclusively by program management 

0 

program management makes decisions but gets inputs from team 

team-program manager relationship adult-adult 

1 

team-program management relationship parent-child 

program management makes decisions but gets inputs from team 

0 

all program team members responsible for program decisions 

when a problem arises: management takes over to solve it 

0 

management lets the team solve the problems 

leadership is do as 1 say, not do as 1 do 

0 

leadership by example 

program expectation not influenced by PM 

0 

program expectation managed by PM 

PM gives freedom to team, but has no mentoring for members (abdication) 

0 

PM empowers teams by mentoring members to be leaders 

promgram management waits and sees what happens then plans 

0 

management plans far in advance 

program management is constantly reacting to emergencies 

0 

management is one step ahead of problems 

facilitative approach to solving problems 

1 

take charge readily and often 

program management is complex, takes much time to understand 

0 

management is simple, easy to figure out 

program management prefers to plunge right in 

0 

takes time to separate things to be done and order of doing them 

program management reacts spur of the moment 

0 

methodically follows plans 

Technical Competency of the Program Manager: 

PM has technical experience particular to the particular s/w program 

1 

PM relies on team members solely 

0 

0 

0 

1 

1 

0 

0 

1 

1 

PM participates in technical reviews 

1 

PM only in non-technical reviews 

PM participates in making technical decisions when problems arise 

1 

PM delegates technical questions 

PM does not get involved discussing technical options 

0 

PM contributes to technical options being discussed 

PM does not review technical options and decisions 

0 

PM reviews technical options and decisions 

PM actively attempts to keep up-to-date with current technology and standards 

1 

PM is removed from cutting edge technology issues 

PM receives technical periodicals and occasionally references applicable articles 

1 

PM doesn't read periodicals nor reference current articles to team 

PM doesn't have technical background (or education) 

0 

PM has technical background (or education) 

team members avoid PM when they need technical advice 

0 

team members generally consider talking to PM regarding technical issues 


HR 



+ Comm. 


□ + Leadership 



+ Tech. Competency 



= People Mgmt. score 



Enter on QMM scoresheet blk c. 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 1 of 2): 


RM is formal and documented 

1 

RM is informal, if at all 

0 

a risk management plan exists 

1 

no risk management plan is developed 

0 

RM is more of a data call than a useful document 

0 

RM drives decisions on the program 

1 

RM is done prior to the program beginning 

0 

RM is done prior and during program execution 

1 

RM is only done during the program execution 

0 

RM is done prior and during program execution 

1 

risks are generalized through the whole program 

0 

risks are categorized 

1 

risk management is done internally, only 

0 

an outside organization also contributes to the RM process 

1 

risk is a management function 

0 

risk is a program team function 

1 

risks are precisely articulated 

1 

risks are generalized, if at all 

0 

each risk has a consequence 

1 

consequences are generalized, if at all 

0 

a mitigation strategy is completed for each risk 

1 

mitigation strategy is generalized, if at all 

0 

contingency plans are developed for a RM plan 

1 

contingency plans are ad hoc as problems arise in the program 

0 

risks are anticipated 

1 

if problems arise, management will deal with it 

0 

the program doesn't have any risk 

0 

programs that do not have risk, have problems 

1 

risk management is automated 

0 

risk management may use tools, but depend on human input 

1 

risks are assigned probabilities 

1 

probabilities are not relevant for RM 

0 

all risks are potential problems, relative priorities for risks are not useful 

0 

risks are weighed relative to other program risks and thus prioritized 

1 

risk management information is only shared internally 

0 

risk management information is shared with all stakeholders 

1 

risk analysis uses ordinal rankings 

0 

risk analysis uses actual measurements with a mathematical model 

1 

regret analysis used 

1 

no regret analysis done 

0 

attach probabilities to future events 

1 

no probabilities associated with future events 

0 

assessing risks with mechanical meethods 

0 

risks should be compared to other risks and sorted 

1 

risk status tracked 

1 

not tracked 

0 

technical risks examined 

1 

no technical risks examined 

0 

process risks examined 

1 

no process risks examined 

0 

product risks examined 

1 

no product risks examined 

0 

stakeholder/user risks examined 

1 

no examination of stakeholder/user risks 

0 

checklists used to identify risks 

1 

no checklists used 

0 

risks are tracked 

1 

no tracking or monitoring of risks 

0 

each risk has an impact 

1 

no impact analysis of risk 

0 

each risk has a mitigation plan 

1 

no individual risk mitigation 

0 

risks monitored by priority 

1 

no special attention to track higher priority risks 

0 

risk assessment is formalized 

1 

no formal risk assessment 

0 

risk control is formalized 

1 

no formal risk control 

0 

integration risks not considered 

0 

integration risks examined 

1 


Risk Management (page 1 of 2) score 
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Pair choice section FOUR: (Risk Management(RM)) choose most applicable term of the two for each row (page 2 of 2): 


risks to cost 

1 

no cost risks examined 

0 

unforeseen risks have occurred in program 

0 

any risk that came up had been identified previously 

1 

personnel risks examined 

1 

no personnel risks examined 

0 

estimation risks examined 

1 

no estimation risks examined 

0 

planning risks examined 

1 

no planning risks examined 

0 

requirements risks examined 

1 

no requirements risks examined 

0 

resource risks examined 

1 

no resource risks examined 

0 

risk management plan updated regularly 

1 

no regular risk management plan updates 

0 

risks charted 

1 

risks not charted 

0 

performance risks examined 

1 

performance risks not examined 

0 

program management self risks examined 

1 

no program management risks examined 

0 

risk from program constraints examined 

1 

no program constraint risks examined 

0 

each category of risks are prioritized 

1 

no prioritization 

0 

each category of risks are evaluated for impact 

1 

no impact analysis performed 

0 

each category of risks have control strategy 

1 

no control strategy 

0 

documentation risks examined 

1 

no documentation risks examined 

0 

regret matrix tracked 

1 

no regret matrix or not tracked 

0 

communication of risk activities are facilitated 

1 

no facilitation or promotion of communication of risk activities 

0 

taxonomy-based questionnaire used to identify risks 

1 

taxonomy-based questionnaire not used 

0 

associated hardware risks examined 

1 

no consideration for hardware risks 

0 

integration risks examined 

1 

integration risks not examined 

0 

communication risks examined 

1 

communication risks not examined 

0 

leadership risks examined 

1 

leadership risks not considered 

0 

risk avoidance considered for certain risks 

1 

risk avoidance not considered for risks 

0 

risk documentation forms used 

1 

no risk documentation forms used 

0 

dependency risks examined 

1 

no dependency risks examined 

0 

alternatives like risk avoidance considered for high risk items 

1 

no consideration of risk avoidance 

0 

documented risk statements use a condition-consequence type format 

1 

condition-consequence of risk statements not clearly defined 

0 

no assignment of ownership of risk mitigation action 

0 

each risk mitigation action is assigned to an individual for resolution 

1 

calculation of risk exposure made (probability X loss, for each risk) 

1 

no risk exposure calculations 

0 

oral communication of risks only 

0 

risks written in a way that communicates nature and status of factors 

1 

triggers used to quantify risk conditions present 

1 

risk conditions present are all subjective 

0 

risk "czar" in program for monitoring risks 

1 

no special positions/responsibilities for risk monitoring 

0 

post-program review completed (scheduled) for unanticipated problems ID 

1 

no post-program reviews completed or scheduled 

0 

no schedule risks examined 

0 

risks to schedule investigated 

1 


Risk Management (pg 2 of 2) score 



+pg 1 score 



= TOTAL SCORE 



Enter on QMM scoresheet blk d. 
















































































































APPENDIX B 


Earned Value Management 

‘Gold Card’ 

Defense Acquisition University 




VARIANCES r evc»ar4e is Pcshve, -rfavoreble 8 Negate* 

Cost Variance CV = BCWP - ACWP CV % = (CV / BCWP) .100 

sch*du * Variance sv = BCWP - bcws sv % = (SV / bcws 1.100 

variance at Completion VAC = BAC - EAC 

PERFORMANCE INCHES FeuomWe is > 1 jO, wrfevo’ebte 3 < 1.0 
Cost Efficiency CPI = BCWP / ACWP 

Schedu e EffeaMcy SPi = BCWP / BCWS 



’ERMINCLOGY 

NCC Segobsted Contact Cos: 

AUW Auhcrized U-priced Woik 

CBE Ccntac: Budget Ee;e 

OT5 0*er Target 3 sad ne 

TAB Total A Iscsie: Budget 

BAC Budge: A: Corrpiebcn 

PMB = e^y~o'a Meaauwned BueKee 

MB Vansgeme't Reserve 

UB wrdorb.oed B.dgef 

CA Cental Account 

WP Wot Package 
PP = tenni»g Pecwge 
BCWS Budgexd CcskforWar* Scheduled 
BCWF Budgeted Zcztkr'/tem Se’fcmtes 
ACWP Actus Cost of '/ton SeHor-ea 
EAC Esa-ate A: Co-plebon 

LRE -atest Reuses Esdmete 
SLPP Summery «e»«i Pa-nrg 2 »:<»:e 
TCPl To Co-pete Pertermence rdex 


C o'trad v ce less peeft 1 tee|s) 

W0H1 extra clue I y approves b-t rwt yet nepebete: 1 sefriized 

S-m of MCC and AUW 

S.m cf CBE an: •ecocraeo y/t~.r 

S.moral cucgea teroar*cr co'teect = VCC, CBE. o* 0T3 

"ctal Budget ter bal cor tact terv ery gven e*e 

Contact ime-:b»ed Budget pier 

B.dgef h bred by <b PH ter u'vc-cws, risk menagemen: 

B*oed y oenned echbe: ncl ye: air* buxd x CA: 

_o*e:t OWES etemert es:gneo :o a srge tecal pontte pan & ccnta 

scope. sened-tei b.ooet 

NesMerr- <je«i-canned echoes after, a CA 

r arte*m CA arbvbes not yet defreo into WPs 

Value or •vk pa*res b be eccomdaned = PLANNED VALUE 

Vsi.e o* «9*k aoeaapfished 

Costo^aor«eccompbhed =ACTUA. COST 

Estmeae erbeel oostfo* fatal ccnrecttenj err f guen level; 
mey te oenereteo by <b. PMO, CCt.li, etc = EAC„, 1(->i 
KrsEAC or EACy 4 

r ar-te-n- actvbes not yet aefreo into CAs 
ERciercy reeded ta- ii-e near b achieve an EAC 


OVERALL STATUS 

% Schedule = (BCWS** / BAC) • 100 
y» Complete = (BCWP c ^ / BAC) • 100 
% Spent = (ACWP c-- / BAC). 100 

ESTIMATE AT COMPLETION • 

EAC = Actuals to Dale ♦ [(Remaining Wort) / |Efficiency Factor)] 

EAC cn = ACWP cu> ♦ [iBAC-BC'WP^gi/CPIcug] = BAC / CPI CUM 

EAC^,. = ACWP*. MlBAC-BCWP^MCPtc^.SPIc^)] 

TO COMPLETE PERFORMANCE INDEX -TCPIl * 

TCPI^ = wore Remaining i Cost Remain ng = |BAC - BCWPcm)/ (EAC - ACWP^) 
• To Deriemn ne E trier TCP* o* EAC: You May Rep ace BAC »Ih TAB 


EVM POLICY: OoDI 50C0.2 Table E3.T2 EVMS n accordance weh ASS .ElA-74* is regured for cost cr 
incentr^e ccntracts. subcontracts, inira-govemment work agreementv & wher agreements valued^S20M |Then-Yr $) 
EVMS contracts > S50M (TY $) reouire that the EVM sysxm be forma y validated by the cogncare contracting officer 
Additions Gudanoe in Defense Acqushion Gj defccck and the Earned Value Management Implementation Guide 
iEVMIG 2005;. EVUS s discouraged on 'rm-Fiied Price. Leve cf Effor. A Time & Material effects regardless of dollar 
value. 

EVM CONTRACTING REQUIREMENTS: 

DFAR Clauses * 252 242-7031 for so: citations and 252242-70C2 for contracts 

Contract Ferormanse Repcrt- Di-MGMT-6l4beA * 5 'ormsa 7/ES I*xanreion Eeselne Saffng & Eipenslicn, 
Integrated Master Schedule - D-MGMT41650 ' 

Integrated Baseine Revew (BR) - Mandatory for a I EVMS corxracts > $2CM 
* See tee EVUIG 2005 fo* CPR am IMS bi omg guoo-ce 

EVM Home Page = https .Vacc.daum lievm 
DAL POC i703| B0S-S2S9 |DSN 555| 
eMai Address 
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APPENDIX C 



A 

SPI 

0.98 

0.02 

1.00 

0.00 

1.00 

0.00 

1.21 

0.21 

1.18 

0.18 

1.15 

0.15 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

105 

1.05 

CPI 

1.09 

0.09 

1.07 

0.07 

1.02 

0.01 

1.24 

0.24 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

113 

1.13 

B 

SPI 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

100 

1 

CPI 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

100 

1 

C 

SPI 

0.99 

0.02 

0.98 

0.02 

0.98 

0.02 

0.98 

0.02 

0.97 

0.03 

0.97 

0.03 

0.96 

0.04 

0.98 

0.03 

0.98 

0.03 

0.98 

0.03 

97.4 

0.97 

CPI 

0.95 

0.05 

0.96 

0.04 

0.96 

0.04 

0.97 

0.03 

1.01 

0.01 

1.01 

0.00 

1.04 

0.04 

1.05 

0.04 

1.05 

0.05 

1.05 

0.04 

100 

1 


Program 

Program 

Program 

Program 

Participant 

ApM 

Ai 

BpM 

Bi 

CpM 

c, 

QMM score 

77 

79 

86 

85 

48 

45 

QMM percent 

77 

79 

86 

85 

48 

45 

Success score 

8 

8 

9 

8 

6 

6 

Mean success 
score (0-10) 

8 

8.5 

6 
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